Map reduce模型
-
Upload
dhlzj -
Category
Technology
-
view
264 -
download
0
description
Transcript of Map reduce模型
![Page 1: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/1.jpg)
《MapReduce模型》
丁海亮 2013-02-27
![Page 2: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/2.jpg)
议程
l 计算模型
l 基础架构
l HDFS
l MapReduce
l 集群部署
![Page 3: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/3.jpg)
典型例子
![Page 4: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/4.jpg)
计算模型
![Page 5: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/5.jpg)
基础架构
![Page 6: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/6.jpg)
基础架构
![Page 7: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/7.jpg)
基础架构
![Page 8: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/8.jpg)
HDFS
![Page 9: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/9.jpg)
读取数据
![Page 10: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/10.jpg)
写入数据
![Page 11: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/11.jpg)
MapReduce
![Page 12: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/12.jpg)
MapReduce
![Page 13: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/13.jpg)
MapReduce
public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context ) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } }
![Page 14: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/14.jpg)
MapReduce
public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } }
![Page 15: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/15.jpg)
MapReduce
public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }
![Page 16: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/16.jpg)
实施流程
硬件选型(商用机)
操作系统选型(Linux)
内核调优
硬盘配置 DataNode无须RAID和LVM
网络配置
Hadoop环境配置
运维支撑 rsync自动化同步配置部署
vm.overcommit_memory
![Page 17: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/17.jpg)
集群部署
![Page 18: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/18.jpg)
集群部署
![Page 19: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/19.jpg)
集群部署
![Page 20: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/20.jpg)
论文参考
![Page 21: Map reduce模型](https://reader036.fdocuments.net/reader036/viewer/2022081720/54c6d86d4a7959b7618b45c3/html5/thumbnails/21.jpg)
Thank You