1

I can read data from S3 location using Spark and Glue without issues but when trying to read with Athena for the same table - getting error when running select * from mytable limit 10;

HIVE_CURSOR_ERROR: Can not read value at 0 in block 0 in file


s3://.../part-00073-123-926b-456-c000.snappy.parquet

What could be the issue and how to fix it?

I tried with: MSCK REPAIR TABLE mytable;

That did not help (getting the same error).

Table create statement is:

CREATE EXTERNAL TABLE `mytable`(
co1 ..,
col2 ..
)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
  's3://.../../'
TBLPROPERTIES (
  'CrawlerSchemaDeserializerVersion'='1.0', 
  'CrawlerSchemaSerializerVersion'='1.0', 
  'UPDATED_BY_CRAWLER'='raw_1', 
  'averageRecordSize'='105', 
  'classification'='parquet', 
  'compressionType'='none', 
  'objectCount'='155', 
  'recordCount'='33459791', 
  'sizeKey'='1738251189', 
  'typeOfData'='file')
Joe
  • 11,983
  • 31
  • 109
  • 183
  • Can you provide table definition, i.e. `SHOW CREATE TABLE you_table_name`, please? – Ilya Kisil Oct 24 '19 at 10:05
  • Provided in a question. – Joe Oct 24 '19 at 10:50
  • It seems to me that you problem relates to different parquet conventions used in Hive and Spark, see [here](https://stackoverflow.com/a/49726124/6147064) for more info. How the files on S3 were created? Also, I see that you used Glue Crawler to define table. Do you know what type of values to expect in your colummns? You removed types of columns, so can't tell for sure, but have you tried what is proposed in [this post](https://forums.aws.amazon.com/thread.jspa?threadID=275809&tstart=0&messageID=836262#836262) – Ilya Kisil Oct 24 '19 at 12:26
  • in the table defintion 'compressionType'='none' while the name of the file has snappy in it. Change one or the other. – Guy Oct 25 '19 at 09:46

0 Answers0