4

I want to understand the filesystem counters in hadoop.

Below are the counters for a job that I ran.

In every job that I run, I observe that the Map file bytes read is like almost equal to the HDFS bytes read. And I observe that the file bytes written by the map is the sum of the file bytes and hdfs bytes read by the mapper. Pls help! Is the same data being read by both local file and hdfs, and both are being written to the local file system by the Map Phase?

                Map

FILE_BYTES_READ 5,062,341,139

HDFS_BYTES_READ 4,405,881,342

FILE_BYTES_WRITTEN 9,309,466,964

HDFS_BYTES_WRITTEN 0

Thanks!

Mahalakshmi Lakshminarayanan
  • 714
  • 2
  • 11
  • 34

2 Answers2

8

So the answer is really that what you are noticing is job specific. Depending on the job the mappers/reducers will write more or less bytes to local file compared to the hdfs.

In your mapper case, you have a similar amount of data that was read in from both local and HDFS locations, there is no problem there. Your Mapper code just happens to need to read about the same amount of data locally as it reads from HDFS. Most of the time the Mappers are being used to analyze an amount of data greater than it's RAM, so it's not surprising to see it possibly writing the data it gets from the HDFS to a local drive. The number of bytes read from HDFS and local are not always going to look like they sum up to the local write size (which they don't even in your case).

Here is an example using TeraSort, with 100G of data, 1 billion key/value pairs.

    File System Counters
            FILE: Number of bytes read=219712810984
            FILE: Number of bytes written=312072614456
            FILE: Number of read operations=0
            FILE: Number of large read operations=0
            FILE: Number of write operations=0
            HDFS: Number of bytes read=100000061008
            HDFS: Number of bytes written=100000000000
            HDFS: Number of read operations=2976
            HDFS: Number of large read operations=0

Things to notice. The number of bytes read and written from the HDFS is nearly exactly 100G. This is because 100G needed to be sorted, and the final sorted files need to be written. Also notice that it needs to do a lot of local read/writes to hold and sort the data, 2x and 3x the amount of data it read!

As a final note, unless you just want to run a job without caring about the result. The amount of HDFS bytes written should never be 0, and yours is HDFS_BYTES_WRITTEN 0

greedybuddha
  • 7,488
  • 3
  • 36
  • 50
2

This is the total counter for HDFS distribution, Map phase and Reduce phase of your job.

These counter, like you observe, are the sum of file bytes and hdfs bytes. Hadoop uses your logical filesystem HDFS to share the resources over the cluster.

Thomas Fritsch
  • 9,639
  • 33
  • 37
  • 49
Giorgio Desideri
  • 169
  • 1
  • 12
  • i ve justed pasted the counters of the map phase. i am interseted in knowing about file bytes read.. i think it is the number of bytes read by the local file system. but why is it almost the same as hdfs bytes read in every job i run? – Mahalakshmi Lakshminarayanan May 19 '13 at 12:05
  • 1
    Because Hadoop tries to optimize for local reads, so it distributes your code to where the data lies. That's why your HDFS reads are local reads. – Thomas Jungblut May 19 '13 at 12:37