Kfs presentation
-
Upload
petrovici-florin -
Category
Technology
-
view
617 -
download
2
description
Transcript of Kfs presentation
![Page 1: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/1.jpg)
KFS aka Kosmos FS
Nicolae Florin Petrovici
![Page 2: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/2.jpg)
2
• C++ programmer, several years of experience
• PHP programmer, several years of experience
• Working at 1&1 since March 2011, the backup dpt.
• Interested in : Linux, opensource,architecture design, Nokia Qt, frameworks, Web 2.0
• Personal projects: Web 2.0 search site (will launch in Feb. 2012), small contributions to opensource
About me
![Page 3: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/3.jpg)
3
What's this all about ?
• KosmosFS – distributed FS, written in C++
• Previously known as Cloudstore
• Developed by Kosmix (American company)
• Opensource http://code.google.com/p/kosmosfs/
• Kosmix was so popular, that it was acquired by Walmart, the largest retail store chain in the world
• Several days experience with the product
![Page 4: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/4.jpg)
4
Sneak peek: A large variety of products inside a Walmart store
![Page 5: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/5.jpg)
5
Fun facts
• KFS is modelled after HDFS (Hadoop Filesystem)
• C++ clone, version 0.5, unstable
• Bindings for Java, Python
• 4 bugs in the issue tracker, one submitted by yours truly
• Doesn't compile with g++ 4.6 (boost regex linking error)
• Has some potential
![Page 6: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/6.jpg)
6
So what's a distributed filesystem?
• As it says, it is a filesystem which is distributed
• Allows access to files from multiple hosts
• Usually, has a custom protocol based on TCP/UDP
• Clients do not have direct access to block storage
• It is NOT a shared filesystem (like NFS)
![Page 7: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/7.jpg)
7
So what's a distributed filesystem?
●Data is usually stored in chunks, across the network on multiple nodes●Chunks are replicated using a replication factor of 2 or 3 (fault-tolerance)●Metadata (the chunk location) is maintained using a metaserver●Usually has some sort of single point of failure●It is the next big thing advertised for handling BigData●Hype started with Google Filesystem and MapReduce
![Page 8: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/8.jpg)
8
Google Filesystem in a pic
Chunks are usually 64 MB
![Page 9: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/9.jpg)
9
Google Filesystem general overview• Cheap commodity hardware
• Nodes are divided in : Master Node, Chunkservers
• Chunkservers store the data already broken up in chunks
• Each chunk is assigned a unique 64 bit label (logical mapping of files to chunks)
• Each chunk is replicated several times
• Master Node: table mappings with the 64 bit labels to chunk locations, locations of the copies of chunks, what processes are reading or writing
• Metadata is kept in memory and flushed from time to time (checkpoints)
• Permissions for modifications are handled by a system of time-limited expiring “leases”
• Lease – finite period of time of ownership on that chunk
• The chunk is then propagated to the chunkservers with the backup copies
• ACK mechanism => operation atomicity
• Single point of Failure: Master Node
• Usespace library, no FUSE
• Copies: HDFS (Java stack), KFS (C++ stack)
![Page 10: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/10.jpg)
10
What were we talking about ?
• Modelled after GFS
• MetaServer, ChunkServer, client library
• ChunkServer stores chunks as files
• To protect agains corruptions, checksums are used on each 64KB block and saved in the chunk metadata
• On reads, checksum verification is done using the saved data
• Each chunk file is named: [file-id].[chunk-id].version
• Each chunk has a 16K header that contains the chunk checksum information, updated during writes
The ChunkServer is only aware of its own chunks.
Upon restart, the metaserver validates the blocks and notifies the chunkserver of any stale blocks (not ownred by any file in the system) resulting in the deletion of those chunks
KFS
![Page 11: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/11.jpg)
11
Features
• Per file Replication (of course)
• Re-replication (if possible)
• Data integrity (checksums)
• File writes (lazy writing or force-flush)
• Stale chunks detection
• Bindings (already mentioned that)
• FUSE (Filesystem in Userspace)
KFS
![Page 12: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/12.jpg)
12
Advantages over HDFS
● File writing HDFS writes to a file once and reads many times. KFS supports seeking in file
and writing multiple times
●Data integrity
After you write to a file, the data becomes visible to other apps when the app closes the file. So, if the process were to crash before closing the file, the data is lost. KFS exposes data as soon as it gets pushed out to the chunkservers. It also has a caching mechanism which can be disabled/enabled on client side
● Data rebalancing
Rudimentary support for automatic rebalancing (The system may migrate chunks from over-utilized nodes to under-utilized nodes)
![Page 13: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/13.jpg)
13
Insert smiley face here
![Page 14: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/14.jpg)
14
Java people – why should you be interested?• KFS can be integrated in the Hadoop chain
• Instructions here: http://code.google.com/p/kosmosfs/wiki/UsingKFSWithHadoop
Still
• Actually, you shouldn't be
Hadoop stack is much better:
- Provides all the necessary tools for mapreduce jobs
- Hadoop also has streaming support => clients can also be written in Python/C
- Pig, Hive and other analyzing frameworks
- HDFS is widely used in conjunction with HBASE
• But in a few years:
• Imagine a world with the C++ equivalent stack:
- KFS – distributed filesystem
- HyperTable – C++ equivalent of Hbase (http://www.hypertable.org/)
- Mapreduce in C++ anyone ?
Major advantage: lower memory footprint, faster loading times etc
![Page 15: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/15.jpg)
15
vs
So it's basically
Moore Lennon
![Page 16: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/16.jpg)
16
Building (for those interested)
• Cmake based system
• Binaries are created inside a “build” directory
• Option to build the JNI/Python bindings and the FUSE filesystem
• Doesn't work with g++ 4.6 (boost_regex linking error)
![Page 17: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/17.jpg)
17
Deployment
• Very engineer-wise:
• You must have ssh passwordless acces on all machines
• Script copies binaries and scripts to all nodes
• Caveat: Doesn't work on different architectures
• Define a machine configuration file
• Define a machines.txt file (names of the nodes from machines.cfg file)
• Deploy: python kfssetup.py -f machines.cfg -m machines.txt -b ../build -w ../webui
• Start: python kfslaunch.py -f machines.cfg -m machines.txt -s
• Stop: python kfslaunch.py -f machines.cfg -m machines.txt -S
• Check status: kfsping -m -s <metaserver host> -p <metaserver port>
• Uninstall: python kfssetup.py -f machines.cfg -m machines.txt -b ../build/bin -U
• Also comes with a Python webserver which calls kfsping in the background for those interested
![Page 18: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/18.jpg)
18
Example config file
[metaserver]
node: machine1
clusterkey: kfs-test-cluster
rundir: /mnt/kfs/meta
baseport: 20000
loglevel: INFO
numservers: 2
[chunkserver_defaults]
rundir: /mnt/kfs/chunk
chunkDir: /mnt/kfs/chunk/bin/kfschunk
baseport: 30000
space: 3400 G
loglevel: INFO
![Page 19: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/19.jpg)
19
Cluster setup
• I proceeded enthusiastically to set up my own KFS cluster out of :
• x86_32 laptop (3GB ram) - masterNode
• x86_32 Netbook (Atom 1.2 Ghz 1gb ram) – chunkServer
• armv7 SheevaPlug (ARM 1.2Ghz, 512MB ram) -chunkServer
Slow compile-time on the SheevaPlug (over 1 hour)
Communication done via wireless
By the way, anyone wants to buy a netbook ?
![Page 20: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/20.jpg)
20
Mounting via FUSE
• If you want to mount via FUSE:
kfs_mount <mount_point> -f (force foreground)
You need to have a kfs.prp file:
• metaServer.name = localhost
• metaServer.port = 20000
Unfortunately, the FUSE mount sometimes produces a segfault in one of the chunkservers. C'est la vie. I already filed a bug for this
![Page 21: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/21.jpg)
21
Accessing via the client library
• Rich undocumented OOP/OOD API
• Main entry point: KfsClientFactory
Something in the lines of:
client = getKfsClientFactory()->GetClient(kfsPropsFile)
client->Mkdirs(dirname);
fd =client->Create(filename.c_str(), numReplicas)
client->SetIoBufferSize(fd, cliBufSize)
client->Write(fd, buffer, sizeBytes)
Other APIs: Enable/Disable Async write, CompareChunkReplicas, GetDirSummary, AtomicRecordAppend
Also available in the Java/Python distribution near you
![Page 22: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/22.jpg)
22
Support opensource• This project needs your help in order to
thrive and to provide a real alternative to the Java stack
• API is good, bindings are ready
• Still has some bugs to be fixed
• Not production ready, maybe in a few years?
![Page 23: Kfs presentation](https://reader034.fdocuments.net/reader034/viewer/2022052522/5479e0cab4af9fc9158b4950/html5/thumbnails/23.jpg)
23