1

I am using spark 2.3 and connecting sparkThrift with beeline.

Hive jdbc version 1.2.1 Spark SQL version 2.3.1

I am trying to create external table with skip header property but select command is always returning data with header as first row, below is my create query

CREATE EXTERNAL TABLE datasourcename11(
`retail_invoice_detail_sys_invoice_no` STRING,
`store_id` STRING,
`retail_invoice_detail_invoice_time` STRING,
`retail_invoice_detail_invoice_date` string,
`cust_id` STRING,
`article_code` INTEGER,
`retail_invoice_detail_base_price` INTEGER,
`retail_invoice_detail_sale_price` INTEGER,
`retail_invoice_detail_quantity` DOUBLE,
`retail_invoice_detail_total_amount` DOUBLE
) 
ROW FORMAT DELIMITED  FIELDS TERMINATED BY ',' 
LINES TERMINATED BY '\n'  
LOCATION '/home/java_services/backend/demo/' 
TBLPROPERTIES('skip.header.line.count'=1);
Jainik
  • 2,352
  • 1
  • 19
  • 27
Mahesh J
  • 94
  • 1
  • 9

1 Answers1

0

This property skip.header.line.count=1 is supported in Hive only.

The workaround is to use filter

retail_invoice_detail_sys_invoice_no!=<col name in header>

leftjoin
  • 36,950
  • 8
  • 57
  • 116