2

I would like to populate a postgres table from an avro file using sqoop (2) export, but i dont have id field in the source and that should be populated automatically (serial type) but i am getting an error.

table DDL:

CREATE TABLE test
 (
 id serial primary key,
 partner_id varchar,
 column1 varchar,
 column2 varchar
)

avro schema:

{
"namespace": "avro_test",
"type": "record",
"name": "test",
"fields": [
      {"name": "partner_id", "type": "string"},
      {"name": "column1", "type": ["string","null"]},
      {"name": "column2", "type": ["string","null"]}
      ]
}

export command i use:

./sqoop-1.4.5.bin__hadoop-2.0.4-alpha/bin/sqoop export \
    --connect jdbc:postgresql://host/db \
    --username user_test --password pass_test \
    --table test \
    --export-dir path \
    --columns partner_id,column1,column2

but i am getting an error that i dont have id in the avro schema:

Status : FAILED
Error: java.io.IOException: Cannot find field id in Avro schema

I tried to specify target columns with --columns argument but it does not work. How i can load the above avro file?

If i remove id field from table it exports successfully

thanks in advance

clairvoyant
  • 129
  • 1
  • 14

1 Answers1

0

Simple solution is to add id in your avro schema, default it will be null

{
  "namespace": "avro_test",
  "type": "record",
  "name": "test",
  "fields": [
        {"name": "id", "type": ["null", "int"]},
        {"name": "partner_id", "type": "string"},
        {"name": "column1", "type": ["string","null"]},
        {"name": "column2", "type": ["string","null"]}
  ]

}

While exporting via Sqoop into mysql, it automatically populates primary key "Id". Hope this helps!

blueberry
  • 325
  • 2
  • 8