2

HDFS has a large block size (64 MB), one of the reasons stated is the seek time becomes higher for HDD with smaller block sizes which is why the block size is larger (The other reason being size of index on name node increases, but lets not discuss that in the answers)

Now when we use SSD is seek time really significant as compared to HDD and does that matter?

seahorse
  • 2,420
  • 4
  • 31
  • 40

1 Answers1

3

You are correct that seek time is not a big of an issue with SSDs around.

In HDDs seek time is typically between 0.2 and 0.8 ms. With SSDs there are no moving parts, so a measurement of the seek time is only testing electronic circuits preparing a particular location on the memory in the storage device. Typical SSDs will have a seek time between 0.08 and 0.16 ms.

But seek time is not the only criteria which decide the block size calculation. HDFS is generally used as the storage FS for distributed systems like Hadoop. Hadoop, and other distributed processing ecosystems, work on the concept of Divide and Conquer. That is divide the task into smaller fragments and process them on multiple machines with commodity hardware.

What will happen if the block size is set to too large having the fact in mind that SSDs have a seek time an order of magnitude less than HDDs?

This essentially means lower parallelism. A task for processing that can be divided among 10 machines for a file of 640 MB with 64 MB block size, may limit to 3 parallel mappers with a block size of 256 MB. This will essentially translate to longer running tasks, which will eventually result in slower jobs. Things will get worse when the task in hand is a compute intensive process.

What will happen if the block size is set to too small having the fact in mind that SSDs have a seek time an order of magnitude less than HDDs?

  1. Now in this case, you will have a lot of tasks since the block size is low and a lot of JVMs will be spawned on your cluster resulting in inefficient utilization of your cluster.
  2. Your NameNode will be swamped with a lot of information with, may be, a lot of blocks created even for a small file. Which may now translate to having a more beefier NameNode or some other solution like a NameNode federarion.

So the crux is yes I have SSDs. But chose your block size appropriately! Else you may end up with a slower cluster with SSD and large block size as opposed to your existing implementation of HDDs and appropriate block size

Hope that helps!

Rahul
  • 2,354
  • 3
  • 21
  • 30