MAX-FILES param in WRITE HDFS component issue

Asked Sep 27 '16 at 11:56

Active Sep 27 '16 at 11:56

Viewed 363 times

I have a hadoop table which is partitioned on event_date and MAX_FILES param is set as 200. Now I am trying to load 180 days data. Technically it should work as it will open 180 files. But it's failing. If I assign a higher value e.g. 300, it is working. What could be the reason?

Any help would be much appreciated!

asked Sep 27 '16 at 11:56

Sonalika

When you say "MAX-FILES", are you refering to the `hive.exec.max.created.files` property? – Jaime Caffarel Sep 27 '16 at 14:19
If you create the table (with a sufficiently high setting) and then count your files, do you find a number between 200 and 300? – Dennis Jaheruddin Sep 27 '16 at 14:22
@JaimeCr .. there is a parameter MAX_FILES in Write HDFS component. I have set this as 200. – Sonalika Sep 28 '16 at 06:04
@Dennis Jaheruddin.. I am actually getting a much higher file count - hdfs dfs -ls /b2b/cs/base/rs_groups_b|wc -l 1075 still, 300 as MAX_FILE parameter is working fine. How is this possible? – Sonalika Sep 28 '16 at 06:06
Are you sure you are counting the location of the table? I would expect it to be somewhere in /apps/hive/warehouse – Dennis Jaheruddin Sep 28 '16 at 09:46
yes. this is the table location. – Sonalika Sep 28 '16 at 10:21

MAX-FILES param in WRITE HDFS component issue

0 Answers0