1

I am new to BigData i don't know what is going on! Please note, I am learning this myself.

I imported a table called sqooptest from MySQL, from a db called sqoopex8 using this command:

sqoop import \
--connect jdbc:mysql://localhost/sqoopex8 \
--table sqooptest \
-m 1

I don't know where it goes (or imports).

There is a bunch of errors that is thrown, and honestly, i don't even know what to look for in the error. If it's the last line, it says "16/04/23 01:46:52 ERROR tool.ImportTool: Error during import: Import job failed!" Again, I am in learning phase and I am learning this all by myself, so please bear with me!

Now, I look under /user/hduser/ and there is a folder by the name of the table (sqooptest). Nothing inside it though.

Next, intuitively, looking around in the Internet, i find out that MySQL saves all its dbs in /var/lib/mysql. Apparently, i didn't have access to it, so I had to access it from the terminal (CLI). When I did, I found all my dbs there. Now, I did this:

sqoop import \
--connect jdbc:mysql://localhost/sqoopex8 \
--table sqooptest \
--target-dir /var/lib/mysql \
-m 1

( added --target-dir /var/lib/mysql)

This worked for some reason. when I do hadoop fs -ls /var/lib/mysql I see two files - _SUCCESS and part-m-00000. Why is that? Why did it not work in the first time.

Also, in the first attempt, even when I specify the target to HDFS --target-dir /user/hduser, it doesn't take it for some reason. When I give the target as a local file system, it takes it. Why?

Anonymous Person
  • 1,437
  • 8
  • 26
  • 47

2 Answers2

0

Sqoop Needs a empty target path to save files. the path you gave /var/lib/mysql is used as the hdfs path to save the imported files.

/user/hduser might not have worked because that directory might exist or you don't have privileges to create one. try hadoop fs -mkdir to check.

For SUCCESS and part files here is the nice post

What are SUCCESS and part-r-00000 files in hadoop

Community
  • 1
  • 1
vgunnu
  • 826
  • 8
  • 6
  • thnx for your response. How can I delete everything sqoop (all directories and files) clear out mysql and start over? Any suggestion? I don't think anything is working out, even in an empty directory – Anonymous Person Apr 22 '16 at 21:46
  • Following options are good. 1. Run sqoop through a script and use hadoop fs -rmr to delete the folder and then sqoop-import 2. Create a hive table and use sqoop hcatalog import to overwrite the table. Here are some examples and documentation https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_dataintegration/content/sqoop-hcatalog-integration.html https://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_sqoop_hcatalog_integration http://stackoverflow.com/questions/17334509/import-from-mysql-to-hive-using-sqoop – vgunnu Apr 25 '16 at 14:05
0

By default sqoop will import the data from RDBMS to hdfs in the hdfs user directory eg: /user/hduser/[tablename]/part-m-00000 To avoid this and to store the data in our desired directory we have to mention it in the clause --target-dir. But this path should be existed except the last directory in the path. like if --target-dir is /user/hduser/mydirectory/mytabledata is the path, /user/hduser/mydirectory/ should exist in the hdfs then sqoop will create a directory[mytabledata here] under /user/hduser/mydirectory/ and will import the rdbms table data in this directory.

sqoop import \ --connect jdbc:mysql://localhost/sqoopex8 \ --table sqooptest \ --target-dir /path/to/your/desireddir \ -m 1

Please check the documentation here .

Ashok
  • 75
  • 2
  • 7