Stack Overflow
Stack Overflow

Questions tagged [sqoop]

Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.

Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.

You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.

Available Sqoop commands:

  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import mainframe datasets to HDFS
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  version            Display version information

Sqoop has been a Top-Level Apache project since March of 2012.

References

Related Tags

2610 questions
4
votes
1 answer

Sqoop import : composite primary key and textual primary key

Stack : Installed HDP-2.3.2.0-2950 using Ambari 2.1 The source DB schema is on sql server and it contains several tables which either have primary key as : A varchar Composite - two varchar columns or one varchar + one int column or two int…
Kaliyug Antagonist
  • 3,512
  • 9
  • 51
  • 103
4
votes
2 answers

How to store password in password file sqoop

I want to store the password into a file & later use the same in sqoop command. According to sqoop documentation --password-file option allow us for storing password. so i am storing it in pwd file with password abc text only. & hits the below…
Bhavesh Gadoya
  • 196
  • 2
  • 13
4
votes
4 answers

How can we automate incremental import in SQOOP?

How can we automate the incremental import in SQoop ? In incremental import, we need to give the --last-value to start the import from the last value onwards, but my job is to frequently import from RDBMS, I don't want to give last value manually,…
Devender Prakash
  • 81
  • 2
  • 8
4
votes
2 answers

sqoop, select specific columns

In the sqoop statements, is there a provision where we can select only specific columns from oracle side? 1 : works sqoop import --target-dir /tmp/customers --query "SELECT * FROM schema1.customers where item>=1234 and \$CONDITIONS" --connect…
sharp
  • 2,140
  • 9
  • 43
  • 80
4
votes
1 answer

[Vertica][VJDBC](100172) One or more rows were rejected by the server

I got the following error when loading data from Impala to Vertica with Sqoop. Error: java.io.IOException: Can't export data, please check failed map task logs at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112) …
yuan0122
  • 441
  • 9
  • 18
4
votes
2 answers

Sqoop: double quotes query

I have a problem with the double quotes on this sqoop query: select i.Number, i.Date,i.Station, i.lStation, count(*) ax, “1- Pd” St , b.Type from Leg jl, yLeg i, senger b, where jl.LegID = i.LegID and jl.rID = b.erID and b.gID = b.ID and b.tus…
José M
  • 41
  • 1
  • 3
4
votes
2 answers

Difference between --fileds-terminated-by & --input-fields-terminated-by in sqoop hive commands

Can anyone tell what is the difference b/w these 2 arguments --fileds-terminated-by & --input-fields-terminated-by in sqoop ? & when to use the same with an example
Priya v v
  • 143
  • 2
  • 3
  • 9
4
votes
5 answers

java.lang.NoSuchMethodError : org.apache.commons.io.FileUtils.isSymLink(Ljava/io/File;)Z

I am getting this error while importing the data using sqoop(master machine) from oracle db which is in different machine(i.e., slave machine). I have replaced the commons.io.jar file also.
Bhargavi Enuturla
  • 41
  • 1
  • 1
  • 3
4
votes
2 answers

Sqoop + Postgresql: how to prevent quotes around table name

I am trying to import a table from Postgresql to a Parquet file on HDFS. Here is what I do: sqoop import \ --connect "jdbc:postgresql://pg.foo.net:5432/bar" \ --username user_me --password $PASSWORD \ --table foo.bar \ --target-dir…
Bartosz Marcinkowski
  • 6,651
  • 4
  • 39
  • 69
4
votes
1 answer

Partitioning in hive

I'm using static partition in hive to seggregate the data into subdirectories based on date field, I'll need 365 partitions/year for each table(total 14 tables) as I have daily loads into hive. Is there any limitation on number of static partitions…
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72
4
votes
2 answers

Multiple table join in hive

I have migrated Teradata tables' data into hive . Now I have to build summary tables on top of imported data. Summary table needs to be built from five source tables If I go with joins I'll need to join five tables is it possible in hive ? or should…
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72
4
votes
1 answer

Exception with sqoop, com.mysql.jdbc.exceptions.jdbc4.CommunicationsException

I am trying to import data from MySQL to HDFS by using sqoop. To do this I am using following command. sqoop import --connect jdbc:mysql://192.168.0.121:3306/snachitdb --username root --password root --table webdemo_vendorinfo --m 1 --target-dir…
Vikram Singh Chandel
  • 1,290
  • 2
  • 17
  • 36
4
votes
2 answers

Sqoop support for json data type for a table field?

I am trying to import tables from Postgresql to HDFS, using Sqoop. It works fine. But when I try to import a table, where one of the fields is of json type, it shows the error: ERROR orm.ClassWriter: Cannot resolve SQL type 1111 It seems like Sqoop…
user1680859
  • 1,160
  • 2
  • 24
  • 40
4
votes
2 answers

How to import data using Sqoop from RDBMS into local file system, not HDFS?

I have seen Sqoop allows one to import data from an RDBMS database into HDFS. I however want to import data into local file system folders in Linux and NOT into HDFS file system. Is there a way to do this?
Syed Nasar
  • 43
  • 1
  • 3
4
votes
4 answers

Differences between Apache Sqoop and Hive. Can we use both together?

What is the difference between Apache Sqoop and Hive? I know that sqoop is used to import/export data from RDBMS to HDFS and Hive is a SQL layer abstraction on top of Hadoop. Can I can use Sqoop for importing data into HDFS and then use Hive for…
Raghavendra Kumar
  • 43
  • 1
  • 2
  • 5
1 2 3
…
99 100