I'm moving the data from Mysql to S3 using data pipeline and it creates empty file for couple of days. I believe, it is making my athena query fails with
"HIVE_CURSOR_ERROR: Unexpected end of input stream".
Below is my script
CREATE EXTERNAL TABLE `test`(
`col0` bigint,
`col1` bigint,
`col2` string,
`col3` string
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
's3://dummy/'
Could you please let me know if there is any option to skip zero bytes S3 file?