3

So - when Tez chooses number of mappers to run, it looks at the number of containers which can run in parallel (available slots), a wave factor, rack locality of data, FileInputFormat max split size, Tez max grouping size, stripes which can go into splits, uncompressed total data size of columns to be fetched etc - it does not look at the tez container size.

So the calculation of number of mappers results in a input slit length bytes per mapper - which can be estimated (before running the job).

But - how to estimate, the total container size needed (memory) to process that input split ?

I understand the memory needed will depend on

  1. Input split length raw (bytes)
  2. Compression (percentage?)
  3. Any UDF which will be applied on the records (negligible probably)
  4. Vectorization if being used (boolean)
  5. Map join if needed (boolean)
  6. Sorting if needed (boolean)
  7. Buffer used before writing into disk (percentage?)

But - how can I estimate the container size or rather the heap space needed within container based on input split bytes ?

One way is to look into committed heap bytes of a mapper task after one run.

But is there any formula to estimate the COMMITTED_HEAP_BYTES from INPUT_SPLIT_LENGTH_BYTES based on the above factors or any other factors ?

Igor Dvorzhak
  • 4,360
  • 3
  • 17
  • 31
Run2
  • 1,839
  • 22
  • 32

1 Answers1

0

I don't think input split length per mapper affects Tez container size directly. It just means the split will be processed by one mapper, but it doesn't mean the whole split will be loaded into memory at once. So the split length could be much larger than the Tez container size which runs the mapper.

As a general guideline,

Set Tez container size to be the same as or a small multiple (1 or 2 times that) of YARN container size yarn.scheduler.minimum-allocation-mb but NEVER more than yarn.scheduler.maximum-allocation-mb. You want to have headroom for multiple containers to be spun up.

See more details in this doc.

Dagang
  • 24,586
  • 26
  • 88
  • 133
  • We have nodes which are 104GB. Yarn can allocate containers from 1GB to 80GB. So the span is huge. Generally tez grouping size for stripes tez.grouping.max-size is 1GB. Additionally if the average file size for tables is around 1GB, then split strategy BI or ETL will both fail on 2GB containers (using the above rule). I have noticed 1GB splits result in approx 4 to 5 GB committed heap bytes on mappers. That is what I am after, a calculation based on the factors I mentioned in my question - or any other factor. – Run2 Oct 15 '20 at 05:03