hadoop - Not getting correct output when running standard "WordCount" program using Hadoop0.20.2 -

-

I'm new to Hadop. I'm trying to run the famous "WordCount" program - the total number of words in a list of files using Hadoop-0.20.2. I am using a single node cluster.

Foling is my program:

Import java.io.File; Import java.io.IOException; Import java.util. *;

Import org.apache.hadoop.fs.Path; Import org.apache.hadoop.conf ; Import org.apache.hadoop.io ; Import org.apache.hadoop.mapreduce *; Import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; Import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; Import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; Import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

Public Class WordCount { 

  Public Static Class Map Mapper and Letting; LongVerbable, Text, Text, Intrabate, and GT; {Private Final Static IntWritable A = New IntWritable (1); Private text word = new text (); Public Zero map (long-term appropriate key, text value, reference reference) throws IOException, interrupted; Expression {string line = value Tutorial (); StringTokenizer Tokenizer = New StringTokenizer (line); While (tokenizer.hasMoreTokens ()) {word.set (tokenizer.nextToken ()); Context.write (word, one); }}} Reducer reduction in public stable class  value, reference reference) throws IOException, interrupted; Exception {int sum = 0; While (values.hasNext ()) {++ sum; } Context.write (Key, New IntWritable (Yoga)); }} Public static zero main (string [] args throws exceptions {configuration conf = new configuration ()); Job job = new job (conf, "wordcount"); FileInputFormat.addInputPath (Job, New Path (Args [0])); FileOutputFormat.setOutputPath (job, new path (args [1])); Job.setJarByClass (WordCount.class); Job.setInputFormatClass (TextInputFormat.class); Job.setOutputFormatClass (TextOutputFormat.class); Job.setMapperClass (Map.class); Job.setMapOutputKeyClass (Text.class); Job.setMapOutputValueClass (IntWritable.class); Job.setReducerClass (Reduce.class); Job.setOutputKeyClass (Text.class); Job.setOutputValueClass (IntWritable.class); Job.setNumReduceTasks (5); Job.waitForCompletion (true); }  
  
 Program using Hoop -0.20.2 (not showing the command for clarity), the incoming output is A1A1B1B! C1C1D! D1  
 which is wrong. The actual output should be: A2B2C2D2  
 This "WordCount" program is a very standard program. I'm not really sure this code is wrong with the content of all configuration files like mapper-site.exml, core-site.exml etc.  
 I would be happy if someone could help me.  
 Thank you.   
 
 
 This code actually runs a local metret job. If you want to submit it to the actual cluster, you must provide the  fs.default.name  and  mapred.job.tracker  configuration parameters. These keys are mapped to a host with your machine: the port pair will just be your Managed / Core-Site. Like in Xml. 
 Ensure that your data is available in HDFS, not on the local disk, as well as the number of reducers should be reduced. It has about 2 records per reducer copy, you should set it to 1.

Comments

Post a Comment

Popular posts from this blog

qt - switch/case statement in C++ with a QString type -

-
I want to use switch-case in my program but the compiler generates an error. How can I use the switch statement with QString ? The compiler gives me this error: The expression of the switch type 'QString' is invalid My code is as follows : boolStopWord (the word QString) {bool flag = false; Switch (word) {case "the": flag = true; break ; "On" case: flag = true; break ; Case "in": flag = true; break ; Case "your": flag = true; break ; Case "near": flag = true; break ; Case "all": flag = true; break ; Case "this": flag = true; break ; } Return flag; } How do I use the switch statement with a Caststring ? You can not use the switch statement in C ++ language with integral or enum types. You can formally enter the square type of object in the switch statement, but this means that the compiler will look for a user-defined conversion to convert it to an integral or enum type.
Read more

python - sqlite3.OperationalError: near "REFERENCES": syntax error - foreign key creating -

-
I'm trying to create a predefined key after checking the manual and I wrote it: Db.execute ("create table" Table 2 (id, integer primary key, somedata integer) "db.execute" (table creation table 2 does not exist (name text, maintain reference (id) ") and found this: sqlite 3. Operation error: near" references ": syntax error Your second ligne is wrong db.execute ("create table if table does not exist 2 (name text, my_id integer, foreign key (my_id) reference Maintenance (ID)) ") as explained
Read more

Python's equivalent for Ruby's define_method? -

-
What is a Python equivalent for Ruby's define_method , which is the dynamic generation of class methods will allow? (As can be seen in Wikipedia) Functions are first-class objects in Python and assign them The properties of a class or an example can be a way of doing the same thing as a Wikipedia example: colors = {"black": "000", "red": "F00", "green": "0f0", "yellow": "ff0", "blue": "00f", "magenta": "fff", "cyan": "0ff", "white": " FFF "} Class MyStream (ARR): Pass for Name, Code in colors.iteritems (): def _in_colour ( W code = code): return '& lt; Span style = "color:% s" & gt;% s & lt; / Span & gt; ' % (Code, self) setter (histrive, "in_" + name, _in_ color)
Read more