Questions tagged [sqoop]

Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.

Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.

You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.

Available Sqoop commands:

  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import mainframe datasets to HDFS
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  version            Display version information

Sqoop has been a Top-Level Apache project since March of 2012.

References

Related Tags

2610 questions
7
votes
3 answers

Sqoop Hive table import, Table dataType doesn't match with database

Using Sqoop to import data from oracle to hive, its working fine but it create table in hive with only 2 dataTypes String and Double. I want to use timeStamp as datatype for some columns. How can I do it. bin/sqoop import --table TEST_TABLE…
anu
  • 71
  • 1
  • 1
  • 3
7
votes
4 answers

Sqoop Hive exited with status 1

When I execute: sqoop import --connect jdbc:mysql://localhost/testdb --table test --hive-table test --hive-import -m 1 I get following error message: 13/04/21 16:42:50 ERROR tool.ImportTool: Encountered IOException running import job:…
Diddy
  • 193
  • 1
  • 3
  • 11
7
votes
6 answers

Sqoop - Could not find or load main class org.apache.sqoop.Sqoop

I installed Hadoop, Hive, HBase, Sqoop and added them to the PATH. When I try to execute sqoop command, I'm getting this error: Error: Could not find or load main class org.apache.sqoop.Sqoop Development Environment: OS : Ubuntu 12.04 64-bit Hadoop…
talha06
  • 6,206
  • 21
  • 92
  • 147
6
votes
3 answers

Where do I install a jdbc driver on ubuntu?

I'm trying to install the MS SQL JDBC driver on ubuntu to be used with sqoop for Hadoop. I'm totally new to java and linux, so I'm not sure where to extract everything to.
Micah
  • 111,873
  • 86
  • 233
  • 325
6
votes
0 answers

sqoop error: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution

hduser@kubuntu17:~$ sqoop version /usr/local/hadoop/libexec/hadoop-functions.sh: line 2326: HADOOP_ORG.APACHE.SQOOP.SQOOP_USER: bad substitution /usr/local/hadoop/libexec/hadoop-functions.sh: line 2421: HADOOP_ORG.APACHE.SQOOP.SQOOP_OPTS: bad…
snag05
  • 101
  • 6
6
votes
1 answer

In sqoop export, Avro table to define schema in RDBMS

I'm loading data from HDFS to mySQL using SQOOP, in this data one record has got more than 70 fields, making it difficult to define the schema while creating the table in RDBMS. Is there a way to use AVRO tables to dynamically create the table with…
Rohit Nimmala
  • 1,459
  • 10
  • 28
6
votes
2 answers

How to overwrite MySQL table when using sqoop export from Hive

I need to transfer data from Hive to MySQL. Here is my sqoop command: jdbc:mysql://mysqlserver --username username --password password --table test --columns "member_id,answer_id,answerer_id" -m 1 --export-dir /user/hive/warehouse/utils.db/test…
Fizzy Chan
  • 141
  • 2
  • 5
6
votes
3 answers

sqoop EXPORT - There is no column found in the target table

I wrote a simple script to create a user (TestV100), create a table (Xy100) in that schema and export a tabl delimited flat file from hadoop to this oracle table. This is the shell script: - ExportOracleTestV100.sh #!/bin/bash # Testing…
rvasu
  • 73
  • 2
  • 6
6
votes
2 answers

Passing parameter to sqoop job

I'm crceating a sqoop job which will be scheduled in Oozie to load daily data into Hive. I want to do incremental load into hive based on Date as a parameter, which will be passed to sqoop job After researching lot I'm unable to find a way to pass a…
Chhaya Vishwakarma
  • 1,407
  • 9
  • 44
  • 72
6
votes
2 answers

Executing Sqoops using Oozie

I have 2 Sqoops that loads data from HDFS to MySQL. I want to execute them using Oozie. I have seen that Oozie is an XML file. How can I configure it so I can execute those Sqoop? Demonstration with steps will be appreciated? Two Sqoops…
Rio
  • 765
  • 3
  • 17
  • 37
6
votes
1 answer

how can i provide password to SQOOP through OOZIE to connect to MS-SQL?

I'm exporting information from HDFS into MS-SQL using SQOOP. I'm running SQOOP through OOZIE. Right now I've hard-coded the uid, pwd for the jdbc connection in the OOZIE workflow. Once I switch to prod I won't be able to do this. What is the…
hba
  • 7,406
  • 10
  • 63
  • 105
6
votes
2 answers

How to find optimal number of mappers when running Sqoop import and export?

I'm using Sqoop version 1.4.2 and Oracle database. When running Sqoop command. For example like this: ./sqoop import \ --fs \ --jt \ …
Bohdan
  • 16,531
  • 16
  • 74
  • 68
6
votes
4 answers

Oozie + Sqoop: JDBC Driver Jar Location

I have a 6 node cloudera based hadoop cluster and I'm trying to connect to an oracle database from a sqoop action in oozie. I have copied my ojdbc6.jar into the sqoop lib location (which for me happens to be at:…
nemo
  • 1,504
  • 3
  • 21
  • 41
5
votes
5 answers

ERROR hive.HiveConfig: Could not load org.apache.hadoop.hive.conf.HiveConf. Make sure HIVE_CONF _DIR is set correctly

I am trying to import data from sqoop to hive MySQL use sample; create table forhive( id int auto_increment, firstname varchar(36), lastname varchar(36), primary key(id) ); insert into forhive(firstname, lastname)…
Programmeur
  • 190
  • 1
  • 14
5
votes
1 answer

Can sqoop run without hadoop?

Just wondering can sqoop run without a hadoop cluster? sort of in a standalone mode? Has anyone tried to run sqoop on spark, please share some experiences on it.
Shengjie
  • 12,336
  • 29
  • 98
  • 139