Write Your 1st Map-Reduce Program
-
Upload
bispsolutions -
Category
Documents
-
view
225 -
download
0
Transcript of Write Your 1st Map-Reduce Program
-
7/30/2019 Write Your 1st Map-Reduce Program
1/17
Write your first Map-ReduceProgram
Agenda
Install Eclipse on Ubuntu.Create a Java project in eclipse.Set Java build path.Add external Hadoop jars.Write WordCount program.Mapper for WordCount program.Reducer for WordCount program.Main method for WordCount program.Input file for WordCount program.
Run your WordCount program.Provide input/output arguments.Verify the output.
Install Eclipse from given site
https://apps.ubuntu.com/cat/applications/eclipse-platform/
-
7/30/2019 Write Your 1st Map-Reduce Program
2/17
Install Eclipse using software center
Verify successful installation
-
7/30/2019 Write Your 1st Map-Reduce Program
3/17
Launch Eclipse from Dash home
Choose your eclipse workspace
-
7/30/2019 Write Your 1st Map-Reduce Program
4/17
Create a Java project in eclipse
-
7/30/2019 Write Your 1st Map-Reduce Program
5/17
Create a package in the project
Create a class in the package
-
7/30/2019 Write Your 1st Map-Reduce Program
6/17
Set Java build path
-
7/30/2019 Write Your 1st Map-Reduce Program
7/17
Set Java build path
-
7/30/2019 Write Your 1st Map-Reduce Program
8/17
Set Java build path
Write your Map-Reduce program
-
7/30/2019 Write Your 1st Map-Reduce Program
9/17
Mapper for WordCount
public static class WordCountMapper extendsMapper
{private final static IntWritable one = new IntWritable(1);private Text word = new Text();public void map(Object key, Text value, Context context) throwsIOException, InterruptedException{StringTokenizer itr = new StringTokenizer(value.toString());while (itr.hasMoreTokens()){word.set(itr.nextToken());context.write(word, one);
}}}
Reducer for WordCount
public static class WordCountReducer
-
7/30/2019 Write Your 1st Map-Reduce Program
10/17
extends Reducer{private IntWritable result = new IntWritable();public void reduce(Text key, Iterable values,Context context) throws IOException, InterruptedException
{int sum = 0;for (IntWritable val : values){sum += val.get();}result.set(sum);context.write(key, result);}}
Main Method
public static void main(String[] args) throws Exception{Configuration conf = new Configuration();if(args.length != 2){System.err.println("Usage: WordCount ");System.exit(2);
}Job job = new Job(conf, "Word Count");job.setJarByClass(WordCount.class);job.setMapperClass(WordCountMapper.class);job.setReducerClass(WordCountReducer.class);job.setOutputKeyClass(Text.class);job.setOutputValueClass(IntWritable.class);FileInputFormat.addInputPath(job, new Path(args[0]));FileOutputFormat.setOutputPath(job, new Path(args[1]));System.exit(job.waitForCompletion(true) ? 0 : 1);}
Input for WordCount program
-
7/30/2019 Write Your 1st Map-Reduce Program
11/17
Run your WordCount program
-
7/30/2019 Write Your 1st Map-Reduce Program
12/17
Provide input/output arguments
-
7/30/2019 Write Your 1st Map-Reduce Program
13/17
Output of WordCount program
-
7/30/2019 Write Your 1st Map-Reduce Program
14/17
Create JAR for your program
-
7/30/2019 Write Your 1st Map-Reduce Program
15/17
Create JAR for your program
-
7/30/2019 Write Your 1st Map-Reduce Program
16/17
Create JAR for your program
-
7/30/2019 Write Your 1st Map-Reduce Program
17/17
Thanks
Contact Point :-
www.bispsolutions.com