-1

What is Distributed Cahce in Hadoop?

How it works?

Could some one give me inline description of it with real time example?
Bala
  • 81
  • 1
  • 3
  • 7
  • 1
    How many questions are you going continue asking about Hadoop without making some research? Read Hadoop: The Definitive Guide or something. – Balduz Jul 31 '14 at 06:59
  • this might help: https://developer.yahoo.com/hadoop/tutorial/module5.html#auxdata – vefthym Jul 31 '14 at 12:37

2 Answers2

0

The distributed cache can contain small data files needed for initialization or libraries of code that may need to be accessed on all nodes in the cluster. Say for example you have to count no of words occurence in a huge set of file. And you have instructed that you have to count every words except these words in a file given say (ignore.csv which is also large file).

Then you read this ignore.csv in distributed cache is setup function of your mapper or reducer depends on your logic and store it in a data structure where you can access each word easily( e.g. HashMap).

This file will read and stored before mapper and reducer of any machine get started and this distributed cache is same for all the machines running in cluster.

I hope you understand now. Please comment your doubts if any.

Manish Verma
  • 771
  • 7
  • 20
0

DistributedCache is a deprecated class in Hadoop. Here is the right way to use

Hadoop DistributedCache is deprecated - what is the preferred API?

DistributedCache copies the files to all the slave nodes. So that access is faster for the MR job running locally. The cache is not in RAM, its just a file system cache in all the local disk volume of all slave nodes

Community
  • 1
  • 1
Prabakaran
  • 128
  • 1
  • 9