Questions tagged [sqoop]

Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.

Sqoop is an open source connectivity framework that facilitates transfer between multiple Relational Database Management Systems (RDBMS) and HDFS. Sqoop uses MapReduce programs to import and export data; the imports and exports are performed in parallel.

You can use Sqoop to import data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS), transform the data in Hadoop MapReduce, and then export the data back into an RDBMS.

Available Sqoop commands:

  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import mainframe datasets to HDFS
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  version            Display version information

Sqoop has been a Top-Level Apache project since March of 2012.

References

Related Tags

2610 questions
0
votes
0 answers

Sqoop import fail throwing Exception

Sqoop import fail throwing Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.util.StringUtils.toLowerCase(Ljava/lang/String;)Ljava/lang/String; Exception in thread "main" java.lang.NoSuchMethodError: …
0
votes
1 answer

sqoop incremantal import error

i want to import last updates on my as400 table with sqoop import incremantal, this is my sqoop command: i'm sure about allvariables, to_porcess_ts it's a string timestamp (yyyymmddhhmmss) sqoop import --verbose --driver $SRC_DRIVER_CLASS…
Zied Hermi
  • 229
  • 1
  • 2
  • 11
0
votes
1 answer

How to use Sqoop incremental lastmodified if source table does not have timestamp column.

How to use Sqoop incremental lastmodified if source table does not have timestamp column or any such date column and client is not allowing to make any changes in source table.Please Guide!
user8167344
  • 353
  • 2
  • 6
  • 17
0
votes
1 answer

Sqoop: How to get counts of records for all tables before running the sqoop import

I am trying to use sqoop to import all tables in a database from SQL relational database into hive. Before doing that, I would like to know how many records are in each of the tables in my source database. This way, I can have how estimations of…
Max577
  • 5
  • 4
0
votes
1 answer

scala/spark script from CLI

I am running scala_script.scala from unix cli and getting error of: enycription key missing, while the same scala code runs fine in spark-shell. The code is used to access a hive table load it in dataframe, process some transformation and then…
0
votes
0 answers

Sqoop changes Date to Long when ingested data is saved as avrodata

Using Oracle to ingest the data to HDFS as avrodata using SQOOP. The date/timestamp column fields are changed to long so the value is getting altered. Example : 28-MAR-18 12.42.06.328000 PM changes to 1523401161454. Any insights on the issue.
Sumit Khurana
  • 159
  • 1
  • 10
0
votes
0 answers

SAP HANA Sqoop Import

I am trying to sqoop import from a HANA view. I have tried many ways and it still persists. Anyone had a similar experience and also please help me figure out if I m missing something: Sqoop Job : sqoop import --driver com.sap.db.jdbc.Driver…
Harsha TJ
  • 264
  • 1
  • 8
0
votes
2 answers

huge difference between mappers execution time

I try to import a table with sqoop; i use 4 mappers. The problem is there's a huge difference between execution time between the mappers. Some less than 10 mints the others is more than one hour. may you explain why? and how to optimize my import?
Zied Hermi
  • 229
  • 1
  • 2
  • 11
0
votes
1 answer

What happens sqoop fails in between proceed of data

What happens when Sqoop import job fails while importing data into RDBMS-HDFS and vice-versa?
Sipra
  • 31
  • 1
  • 6
0
votes
1 answer

How can I use composite keys in "sqoop export" to update the records?

I'm using sqoop export command to upsert(update and insert) records in Oracle SQL server from hive. For this, I'm using the following command in hadoop: hadoop jar sqoop-job-launcher-0.0.1.jar com.myclass.sqoop.SqoopLauncher sqoop export --connect…
Adarsh Kumar
  • 113
  • 2
  • 15
0
votes
1 answer

Importing data from MYSQL to HIVE using SQOOP

I am trying to copy data from MySQL to HIVE using SQOOP. However, I am getting below error even though I have set HIVE_CONF_DIR variable in bashrc file. Below is my code: sqoop import --connect jdbc:mysql://localhost/sampleOne \ --username root…
RushHour
  • 494
  • 6
  • 25
0
votes
1 answer

Sqoop import always tries to connect to local mysql even when remote server IP is provided in connect parameter

Sqoop import always tries to connect to local mysql even when remote server IP is provided in connect parameter I tried below connect using MySQL to remote MySQL server and it is working fine. I am able to explore the tables. mysql -u anuser -h…
0
votes
1 answer

Copying data from HDFS to hive using SQOOP

I want to copy data from HDFS to hive table. I tried below code but it doesn't throw any error and data is also not copied in mentioned hive table. Below is my code: sqoop import --connect jdbc:mysql://localhost/sampleOne \ --username root…
RushHour
  • 494
  • 6
  • 25
0
votes
1 answer

Number of mappers in sqoop

I know sqoop has an option where we can set number of mappers(default is 4). In real-time projects who decides and how is the no of mappers decided? Do we use default or any arbitrary number? I know some theoretical links which say number of mappers…
0
votes
1 answer

Sqoop incremental SemanticException No files matching hdfs://host/user/xyz/_sqoop/ error?

I am trying to create a saved sqoop job ( incremental ) and use sqoop metastore as repository. ( for security reasons "xyz" being the unix user who is running the job and sample_table is used for reference) sqoop job…
StrangerThinks
  • 246
  • 4
  • 14
1 2 3
99
100