Sqoop import multiple tables but not all

Question

All the searches I've found show how to import one table or recommend the import-all-tables. What if I want 35 of 440 tables from my db. Can I just write one command and separate the tables by comma or do I have to put it in a script and copy and past the commands over and over and change the table name each time?

What I want to do:

echo "Sqoop Import"
--options-file ${path} 
--table tbl1,tbl2,tbl3\
--target-dir ${path}
--m 1\

What I fear I may have to do:

echo "Sqoop Import"
--options-file ${path} 
--table tbl1\
--target-dir ${path}
--m 1

wait 

echo "Sqoop Import"
--options-file ${path} 
--table tbl2\
--target-dir ${path}
--m 1

This will help you : http://stackoverflow.com/questions/34966626/how-to-import-the-data-from-multiple-source-thru-sqoop/34985029#34985029 — vinayak_narune, Apr 08 '16 at 06:38
@vinayak_narune can you post some text explaining the link and I will accept it as the answer. hrobertv answer is not the best as per my comment I left. — AM_Hawk, Apr 11 '16 at 15:41

score 1 · Answer 1 · answered Apr 08 '16 at 18:10

1

Apparently a sqoop developer thought like you. :)

You can use import-all-tables.
And then add --exclude-tables Comma separated list of tables to exclude from import process.

https://sqoop.apache.org/docs/1.4.0-incubating/SqoopUserGuide.html#id1766722

answered Apr 08 '16 at 18:10

hrobertv

158
1
8

You can also use the -m 1. Guessing the table doesn't have a PK. – hrobertv Apr 08 '16 at 18:11
The only problem that I see and have with import-all-tables is that you cannot specify a schema. It wants to bring in all the schemas and their tables. At least I don't know of a way to specify a schema. So I'm left doing what you are originally trying to avoid. I have to list them out. – hrobertv Apr 08 '16 at 18:12
1

also with the answer you provided this is still going to be labour intensive as I have to specify the 405 tables I want to exclude...It would have been nice to have an option like --include-tables then list out the tables you want to include... – AM_Hawk Apr 11 '16 at 15:39

score 1 · Answer 2 · answered Mar 19 '19 at 06:31

1

Use --exclude-tables "table1,table2" option to ignore the table1 and table2.

Do NOT add white-space between the table names (aka. "table1, table2")

answered Mar 19 '19 at 06:31

mingchau

450
3
12

score 0 · Answer 3 · edited Oct 28 '21 at 13:53

0

You can also use the apply the same command for Hive import as:

sqoop import-all-tables \
--connect jdbc:mysql://your_ip_address:3306/database_name \
--driver com.mysql.jdbc.Driver \
--username root \
--warehouse-dir temp_dir_for_staging \
--hive-import \
--hive-overwrite \
--hive-database hive_db \
--exclude-tables list_of_tables_to_be_excluded \
-m 1

Remember in Hive you need staging area.

edited Oct 28 '21 at 13:53

Purushothaman Srikanth

711
4
9
22

answered Aug 17 '17 at 07:28

satish silveri

358
3
17

score 0 · Answer 4 · edited Nov 03 '21 at 10:28

0

sqoop import-all-tables \
--connect jdbc:mysql://localhost/sqoop \
--username root \
--password hadoop \
--warehouse-dir /Sqoop21/AllTables \
--exclude-tables table1,tables2

edited Nov 03 '21 at 10:28

Purushothaman Srikanth

711
4
9
22

answered Mar 23 '21 at 07:09

Shyam Gupta

489
4
8

When importing more than one table using `sqoop-import` command, isn't it a good practice / mandatory to use `--warehouse-dir` instead of `--target-dir` ? – Purushothaman Srikanth Oct 28 '21 at 08:41

Sqoop import multiple tables but not all

4 Answers4