0

I have an EMR setup with 4 r3.4Xlarge machines (total of 128GB (32G/Node) and 1000GB(250GB) SSD is allocated to alluxio).

I have loaded around 650GB of ORC data. But I can see 3 workers have used 80% + space allocated but one of the worker have only used 1%.

Any way to evenly distribute the data across all workers ?

Thanks in advance

AAudibert
  • 1,223
  • 11
  • 23
Rijo Joseph
  • 1,375
  • 3
  • 17
  • 33

1 Answers1

2

Typically, when Alluxio clients read data from the UFS, the client will cache the data to the local worker. If there is a large imbalance in the distribution of the data, then it may indicate that the task distribution is not even.

There is an Alluxio client configuration parameter which can change the default behavior when caching data into Alluxio. For example, you can set:

alluxio.user.file.write.location.policy.class=alluxio.client.file.policy.RoundRobinPolicy

to change the write location policy to round robin, which will distribute the data across the workers more evenly. This configuration parameter will have to be updated on the Alluxio client, which is dependent on the specific framework you are using.

Gene Pang
  • 231
  • 1
  • 5