Import the all tables from RDBMS using sqoop

Question

I am trying to import data from testing mysql database to hadoop using sqoop. But in some tables having primary and some tables does not have primary key.

$sqoop import-all-tables --connect jdbc:mysql://192.168.0.101/mysql -username test -P --warehouse-dir /home/user_all_tables

17/08/01 22:46:54 ERROR tool.ImportAllTablesTool: Error during import: No primary key could be found for table general_log. Please specify one with --split-by or perform a sequential import with '-m 1'.

Kindly suggest me how to use split by in sqoop command line.

score 1 · Answer 1 · answered Aug 02 '17 at 03:27

1

For the import-all-tables tool to be useful, the following conditions must be met:

Each table must have a single-column primary key.
You must intend to import all columns of each table.
You must not intend to use non-default splitting column, nor impose any conditions via a WHERE clause.

Default option doesn't fit with the non primary key table therefore it is not working. Here I will suggests to use -m 1 option to strict the import with one mapper only.

Sqoop command:

import-all-tables --connect jdbc:mysql://192.168.0.101/mysql -username test \ 
-P --warehouse-dir /home/user_all_tables -m 1

answered Aug 02 '17 at 03:27

Sandeep Singh

7,790
4
43
68

Thank You Sandeep – DevPerson Aug 02 '17 at 04:11
Instead of setting -m 1 single mapper you can try another option. use the --spilt-by and specify the column with unique values. – Kumar Aug 02 '17 at 18:22
@kumsgs - yes! but in that case he has to investigate all non primary key tables to find out the integer column to specify with `--split-by` – Sandeep Singh Aug 03 '17 at 02:30
and then individual Sqoop command is required for all non primary key table because integer column could be different for tables. – Sandeep Singh Aug 03 '17 at 02:59

Import the all tables from RDBMS using sqoop

1 Answers1