I have been working on PageRank algorithm with help of Map Reduce jobs.

I need to create Mapper and Reducer classes with the help of which I will be creating jar file.

I am using jar file to work with Hadoop clusters.

Currently my java files is PageRank.java

import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import java.util.ArrayList;
import java.util.StringTokenizer;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;


public class PageRank {
    
    public class PageRankMapper extends Mapper<LongWritable, Text, Text, Text> 
    {
        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException
        {

            String line = value.toString();
            line = line.replaceAll("\\s ",";");
            StringTokenizer token = new StringTokenizer(line, ";");
            ArrayList<String> list = new ArrayList<String>();
            while (token.hasMoreTokens()) 
            {
                    list.add(token.nextToken());
            }
            int size = list.size();
            double ipr = Double.parseDouble(list.get(size-1)); //initial page rank
            String pageid = list.get(0); // pageid is always first element
            int numlinks = (size-2); // number of output links = size  - (first and last)
            double opr = ipr/(double)numlinks;  //output page rank = total/number of links
            String oprtext = (pageid   " "  String.valueOf(opr));
            int loop = 1;

            String outputlinks = "";

            while(loop <= numlinks)
            {

                   outputlinks  = (list.get(loop) " ");

                   context.write(new Text(list.get(loop)), new Text(oprtext));
                   loop  ;
            }

            context.write(new Text(pageid), new Text(outputlinks));
        }
    }

    public class PageRankReducer extends Reducer<Text, Text, Text, Text>
    {

        public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException
        {

                float fpr = 0.0f;
                String outlinks ="";

                for (Text value : values)
            {

                        String line = value.toString();

                        if (line.matches(".*\\d.*"))
                {

                                String[] token = line.split(" ");
                                fpr  = Float.parseFloat(token[1]);
                        } 
                else 
                {

                              outlinks  = line;
                        }
                }

                String output = (outlinks  String.valueOf(fpr));
                context.write(key, new Text(output));

        }
    }        
    public static void main(String[] args) throws Exception {
                int i = 0;
                String opfile = "/part-r-00000";
                Path opath = new Path(opfile);
                Path inputPath = new Path(args[0]);
                Path outputPath = new Path(args[1] );
                while (i < 3)
        {

                        String suffix = ("/" String.valueOf(i));
                        Path spath = new Path(suffix);
                        Job job = getNewJob(i);
                        i  ;
                        job.setJarByClass(PageRank.class);
                        FileInputFormat.addInputPath(job, inputPath);
                        FileOutputFormat.setOutputPath(job, outputPath);
                        job.setMapperClass(PageRankMapper.class);
                        job.setReducerClass(PageRankReducer.class);
                        job.setOutputKeyClass(Text.class);
                        job.setOutputValueClass(Text.class);
                        inputPath = Path.mergePaths(outputPath,opath);
                        outputPath = Path.mergePaths(outputPath,spath);
                        job.waitForCompletion(true);
                }

        }
        private static Job getNewJob(int i) throws IOException 
    {
                Job job = new Job();
                job.setJobName("Page Rank "  String.valueOf(i));
                return job;
 

   }

}

What my directory looks like

hduser@ajyj:~/pagerank$ ls -l
total 12
drwxr-xr-x 2 root root   4096 Oct 24 12:37 input_data
drwxr-xr-x 2 root root   4096 Oct 24 12:38 pagerank_classes
-rw-r--r-- 1 root hadoop 3824 Oct 24 13:39 PageRank.java

I have created hdfs directory.

As of now I am following some tutorials for configuring. https://www.youtube.com/watch?v=6sK3LDY7Pp4

Error on which I received is for this command.

 hduser@ajyj:~/pagerank$ javac -classpath `hadoop classpath` -d '/home/hduser/pagerank/pagerank_classes' '/home/hduser/pagerank/PageRank.java'
/home/hduser/pagerank/PageRank.java:17: error: error while writing PageRank.PageRankMapper: /home/hduser/pagerank/pagerank_classes/PageRank$PageRankMapper.class (Permission denied)
    public class PageRankMapper extends Mapper<LongWritable, Text, Text, Text> 
           ^
Note: /home/hduser/pagerank/PageRank.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
1 error

Any solution for above problem....

CodePudding user response：

Here you have permission denied error message;

error while writing PageRank.PageRankMapper: 
/home/hduser/pagerank/pagerank_classes/PageRank$PageRankMapper.class (Permission denied)

pagerank_classes is owned by root user and only root user can write;

drwxr-xr-x 2 root root   4096 Oct 24 12:38 pagerank_classes

But you are running your command as hduser.

I think if you change file permission (or owner) recursively for the path /home/hduser/pagerank/pagerank_classes/PageRank$PageRankMapper.class to provide write access for your current user, your problem will be fixed.