Accessing files in hadoop distributed cache

Question

I want to use the distributed cache to allow my mappers to access data. In main, I'm using the command

DistributedCache.addCacheFile(new URI("/user/peter/cacheFile/testCache1"), conf);

Where /user/peter/cacheFile/testCache1 is a file that exists in hdfs

Then, my setup function looks like this:

public void setup(Context context) throws IOException, InterruptedException{
    Configuration conf = context.getConfiguration();
    Path[] localFiles = DistributedCache.getLocalCacheFiles(conf);
    //etc
}

However, this localFiles array is always null.

I was initially running on a single-host cluster for testing, but I read that this will prevent the distributed cache from working. I tried with a pseudo-distributed, but that didn't work either

I'm using hadoop 1.0.3

thanks Peter

possible duplicate of [Files not put correctly into distributed cache](http://stackoverflow.com/questions/12708947/files-not-put-correctly-into-distributed-cache) — kabuko, Jan 21 '13 at 23:15

score 35 · Accepted Answer · answered Dec 09 '12 at 02:27

Problem here was that I was doing the following:

Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
DistributedCache.addCacheFile(new URI("/user/peter/cacheFile/testCache1"), conf);

Since the Job constructor makes an internal copy of the conf instance, adding the cache file afterwards doesn't affect things. Instead, I should do this:

Configuration conf = new Configuration();
DistributedCache.addCacheFile(new URI("/user/peter/cacheFile/testCache1"), conf);
Job job = new Job(conf, "wordcount");

And now it works. Thanks to Harsh on hadoop user list for the help.

Harsh is indeed a godsend! .. Wasted quite a number of hours with the same. Thanks! — Shatu, Oct 25 '13 at 22:03

score 11 · Answer 2 · edited Feb 11 '13 at 10:17

11

Configuration conf = new Configuration();  
Job job = new Job(conf, "wordcount");
DistributedCache.addCacheFile(new URI("/userpetercacheFiletestCache1"),job.getConfiguration());

You can also do it in this way.

edited Feb 11 '13 at 10:17

Rushi

4,553
4
33
46

answered Feb 11 '13 at 09:53

StarScream

223
2
12

score 4 · Answer 3 · edited Jun 06 '13 at 09:22

Once the Job is assigned to with a configuration object, ie Configuration conf = new Configuration();

Job job = new Job(conf, "wordcount");

And then if deal with attributes of conf as shown below, eg

conf.set("demiliter","|");

or

DistributedCache.addCacheFile(new URI("/user/peter/cacheFile/testCache1"), conf);

Such changes would not be reflected in a pseudo cluster or cluster how ever it would work with local environment.

score 2 · Answer 4 · edited Nov 16 '14 at 21:40

This version of code ( which is slightly different from the above mentioned constructs) has always worked for me.

//in main(String [] args)
Job job = new Job(conf,"Word Count"); 
...
DistributedCache.addCacheFile(new URI(/user/peter/cacheFile/testCache1), job.getConfiguration());

I didnt see the complete setup() function in Mapper code

public void setup(Context context) throws IOException, InterruptedException {

    Configuration conf = context.getConfiguration();
    FileSystem fs = FileSystem.getLocal(conf);

    Path[] dataFile = DistributedCache.getLocalCacheFiles(conf);

    // [0] because we added just one file.
    BufferedReader cacheReader = new BufferedReader(new InputStreamReader(fs.open(dataFile[0])));
    // now one can use BufferedReader's readLine() to read data

}

Thanks @Somum, it worked for me. I checked with hadoop 1.2.1 — Jagadish Talluri, Nov 18 '15 at 22:00

Accessing files in hadoop distributed cache

4 Answers4

Linked