0

I'm using Sqoop2 (Sqoop 1.99.3-cdh5.1.0) to import data from a postgresql database. The job successfully completes and creates text files in HDFS. The output files are CSV with single quotes, I would like to configure the output to be tab separated without quotes.

Is output format of Sqoop2 configurable?

Vander Rice
  • 161
  • 1
  • 10
  • --fields-terminated-by Sets the field separator character --lines-terminated-by Sets the end-of-line character --mysql-delimiters Uses MySQL’s default delimiter set: fields: , lines: \n escaped-by: \ optionally-enclosed-by: ' – resultsway Aug 06 '14 at 19:47
  • @purpletech those options cannot be provided in the sqoop2 cli – Vander Rice Aug 07 '14 at 01:02

2 Answers2

1

It appears that Sqoop2 does not yet allow output formatting to be configured,

Sqoop allows configuration of the output format with the command line arguments here: http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_large_objects (also thx @purpletech for the paste in comments)

For Cloudera Hadoop users: Sqoop is installed alongside Sqoop2 and can be found where CDH is installed. Hue provides a web interface for Sqoop2 and Sqoop1 must be invoked from the command line.

Vander Rice
  • 161
  • 1
  • 10
0

I had same problem, so I ended up with Sqoop1. Sqoop2 is great but has some disadvantages like:

  • you can not plan Sqoop2 jobs in Oozie, so you can run it manually only.
  • you can not load data directly into Hive or HBase, only to file.
  • you can not configure output delimiters and enclosures.

So I recommend you to use sqoop1, it is quite easy :

sqoop import --connect xxxx --username xxxx --password xxxx --query select * from xxx --target-dir /tmp/xxx -m 1 --fields-terminated-by | --enclosed-by \0

If you are using Hue jobs, it is better not to write this command into Command field of Sqoop job, but insert every command into separate Params (like first param is import, second --connect, third connection string then --username etc.).

Hope it helps and good luck

Radek Tomšej
  • 490
  • 4
  • 15