MapReduce package error

Question

//what Packages needed for these 3 programs??? how to combine this 3 program into one program?? how to do mapreduce using this 3 program in eclipse??

please help me to run this program successfully

os : linux

Exception Faced :

The method TryParseInt(String) is undefined for the type MaxPYear.MaxPubYearReducer

2.The method setInputFormatClass (Class) in the type Job is not applicable for the arguments (Class)

Mapper code :

public static class MaxPubYearMapper extends Mapper<LongWritable , Text, IntWritable,Text>
    {
        public void map(LongWritable key, Text value , Context context)

                throws IOException, InterruptedException 
                {
            String delim = "\t";
            Text valtosend = new Text(); 
            String tokens[] = value.toString().split(delim);
            if (tokens.length == 2)
            {
                valtosend.set(tokens[0] + ";"+ tokens[1]);
                context.write(new IntWritable(1), valtosend);
            }

                }       
    }

Reducer Code :

public static class MaxPubYearReducer extends Reducer<IntWritable ,Text, Text, IntWritable>
    {

        public void reduce(IntWritable key, Iterable<Text> values , Context context) throws IOException, InterruptedException
        {
            int maxiValue = Integer.MIN_VALUE;
            String maxiYear = "";
            for(Text value:values)          {
                String token[] = value.toString().split(";");
                if(token.length == 2 && TryParseInt(token[1]).intValue()> maxiValue)
                {
                    maxiValue = TryParseInt(token[1]);
                    maxiYear = token[0];
                }
            }
            context.write(new Text(maxiYear), new IntWritable(maxiValue));
        }
    }

Driver Code :

public static void main(String[] args) throws Exception  {
        Configuration conf = new Configuration(); 
        Job job = new Job(conf , "Frequency`enter code here`");
        job.setJarByClass(MaxPubYear.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        job.setMapperClass(FrequencyMapper.class);
        job.setCombinerClass(FrequencyReducer.class);
        job.setReducerClass(FrequencyReducer.class);


        job.setOutputFormatClass(TextOutputFormat.class);
        job.setInputFormatClass(TextInputFormat.class);

        FileInputFormat.addInputPath(job,new Path(args[0]));
        FileOutputFormat.setOutputPath(job,new Path(args[1]+ "_temp"));
        int exitCode = job.waitForCompletion(true)?0:1; 

        if (exitCode == 0 )
        {
            Job SecondJob = new Job(conf, "Maximum Publication year");
            SecondJob.setJarByClass(MaxPubYear.class);

            SecondJob.setOutputKeyClass(Text.class);
            SecondJob.setOutputValueClass(IntWritable.class);

            SecondJob.setMapOutputKeyClass(IntWritable.class);
            SecondJob.setMapOutputValueClass(Text.class);

            SecondJob.setMapperClass(MaxPubYearMapper.class);               
            SecondJob.setReducerClass(MaxPubYearReducer.class);

            FileInputFormat.addInputPath(SecondJob,new Path(args[1]+ "_temp"));
            FileOutputFormat.setOutputPath(SecondJob,new Path(args[1]));
            System.exit(SecondJob.waitForCompletion(true)?0:1);                 

        }
    }

I'm unsure what your problem is: the title mentions an error: what error are we talking about? — ljgw, Aug 29 '14 at 11:11
I want to write these above 3 codes into one program and create a jar file. i don't know how to do it and what packages are needed. — Apk, Sep 02 '14 at 05:56

Antariksha Yelkawar · Answer 1 · 2014-09-02T06:28:29.920

just Write them together in one class

required packages are:

package org.myorg;

import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
import java.io.DataInput;
import java.io.DataOutput;

there might be some extras here since I copied them from my code.

package org.myorg;

import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;

public class <your classname as well as filename> { 
    public static class MaxPubYearMapper extends Mapper<LongWritable , Text, IntWritable,Text>
    {
        public void map(LongWritable key, Text value , Context context)

                throws IOException, InterruptedException 
                {
            String delim = "\t";
            Text valtosend = new Text(); 
            String tokens[] = value.toString().split(delim);
            if (tokens.length == 2)
            {
                valtosend.set(tokens[0] + ";"+ tokens[1]);
                context.write(new IntWritable(1), valtosend);
            }

                }       
    }

    public static class MaxPubYearReducer extends Reducer<IntWritable ,Text, Text, IntWritable>
    {

        public void reduce(IntWritable key, Iterable<Text> values , Context context) throws IOException, InterruptedException
        {
            int maxiValue = Integer.MIN_VALUE;
            String maxiYear = "";
            for(Text value:values)          {
                String token[] = value.toString().split(";");
                if(token.length == 2 && TryParseInt(token[1]).intValue()> maxiValue)
                {
                    maxiValue = TryParseInt(token[1]);
                    maxiYear = token[0];
                }
            }
            context.write(new Text(maxiYear), new IntWritable(maxiValue));
        }
    }
    public static void main(String[] args) throws Exception  {
        Configuration conf = new Configuration(); 
        Job job = new Job(conf , "Frequency`enter code here`");
        job.setJarByClass(MaxPubYear.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        job.setMapperClass(FrequencyMapper.class);
        job.setCombinerClass(FrequencyReducer.class);
        job.setReducerClass(FrequencyReducer.class);


        job.setOutputFormatClass(TextOutputFormat.class);
        job.setInputFormatClass(TextInputFormat.class);

        FileInputFormat.addInputPath(job,new Path(args[0]));
        FileOutputFormat.setOutputPath(job,new Path(args[1]+ "_temp"));
        int exitCode = job.waitForCompletion(true)?0:1; 

        if (exitCode == 0 )
        {
            Job SecondJob = new Job(conf, "Maximum Publication year");
            SecondJob.setJarByClass(MaxPubYear.class);

            SecondJob.setOutputKeyClass(Text.class);
            SecondJob.setOutputValueClass(IntWritable.class);

            SecondJob.setMapOutputKeyClass(IntWritable.class);
            SecondJob.setMapOutputValueClass(Text.class);

            SecondJob.setMapperClass(MaxPubYearMapper.class);               
            SecondJob.setReducerClass(MaxPubYearReducer.class);

            FileInputFormat.addInputPath(SecondJob,new Path(args[1]+ "_temp"));
            FileOutputFormat.setOutputPath(SecondJob,new Path(args[1]));
            System.exit(SecondJob.waitForCompletion(true)?0:1);                 

        }
    }
}

Hi Antariksha, Am new to java and I find it difficult to combine all the three codes into one...can you show me how to put all them together in one class? — Apk, Sep 02 '14 at 05:53
edited my answer. also you need add the jar file hadoop-core-*.jar to the classpath — Antariksha Yelkawar, Sep 02 '14 at 06:30
I tried and found some errors. I have made a screen shot of it. can u tell me how to add a screen shot here? — Apk, Sep 03 '14 at 10:38
I have no idea. why dont you just paste the exception by editing the question. also try dragging it here — Antariksha Yelkawar, Sep 03 '14 at 10:58

MapReduce package error

1 Answers1