0

I want to create a Hive table by importing data from MySQL. The following command can create the table - 

sqoop import \
-D mapred.job.name=name \
-Dorg.apache.sqoop.splitter.allow_text_splitter=true \
-connect "connection_detail" \
-username "username" \
-password "pwd" \
-query "SELECT * FROM schema.tablename WHERE  \$CONDITIONS " \
-null-string '' \
-null-non-string '' \
-fields-terminated-by ',' \
-m 1 \
-hive-overwrite \
--hive-import \
--hive-table tablename \
-map-column-hive z_column=bigint,y_column=int,x_column=int \
-delete-target-dir \
-target-dir  "path_dir"

I now want to create dynamic partition columns in the table - I tried the following command. But, it is giving error mentioned below - 

sqoop import \
-D mapred.job.name=name \
-Dorg.apache.sqoop.splitter.allow_text_splitter=true \
-connect "connection_detail" \
-username "username" \
-password "pwd" \
-query "SELECT * FROM schema.tablename WHERE  \$CONDITIONS " \
-null-string '' \
-null-non-string '' \
-fields-terminated-by ',' \
-m 1 \
-hive-overwrite \
--hive-import \
--hive-table tablename \
-map-column-hive z_column=bigint,y_column=int,x_column=int \
--hive-partition-key a_column,b_column,c_column \
--hive-partition-value 'a_data','b_data','c_data' \
-delete-target-dir \
-target-dir  "path_dir"

FAILED: ParseException line 1:367 cannot recognize input near ',' 'b_column' ',' in column type

Where exactly is the error?

TeeKay
  • 1,025
  • 2
  • 22
  • 60
  • Please refer this - https://stackoverflow.com/questions/46263147/does-sqoop-support-dynamic-partitioning-with-hive – sangam.gavini Oct 01 '19 at 10:57
  • @sangam.gavini --create-hcatalog-table only created metadata. How to import the data into the table? – TeeKay Oct 01 '19 at 13:12
  • As per the sqoop user guide i found the information as " If the option --hive-partition-key is specified, then the value of this option is used as the partitioning key for the newly created table. Only one partitioning key can be specified with this option ", so you can try giving only one partition key. And the error is you got is also pointing at the column2 - b_column. – sangam.gavini Oct 01 '19 at 15:38
  • Another work around you can do is get the data to the hdfs first without any partitions and create a table1 on top of it and also create a table2 as per your partitioning requirement and load tabl1 from table2. – sangam.gavini Oct 01 '19 at 16:01
  • @sangam.gavini any example for this? thanks. – user1447718 Aug 14 '20 at 18:31

0 Answers0