Stack Overflow
Stack Overflow

Questions tagged [hive-serde]

SerDe is short for Serializer/Deserializer, an interface used by Hive for both serialization and deserialization during IO and also interpreting the results of serialization as individual fields. A SerDe allows Hive to read in data from a table, and write it back out to HDFS in any custom format. Anyone can write their own SerDe for their own data formats.

Official documentation page: SerDe

There are many SerDe bundled with Hive as well as third-party SerDe, such as:

  • LazySimpleSerDe
  • OpenCSVSerDe
  • RegexSerDe
  • JsonSerDe
  • AvroSerDe
  • ParquetHiveSerDe
  • OrcSerDe
  • MultiDelimitSerDe
164 questions
1
vote
1 answer

How do I remove text from columns in HIVE (sql)

I am trying to import data from a CSV file (latlong.csv) and I want to remove all of the quotes from my columns. Please Refer to first image. First image This is the code I used to import the data CREATE TABLE IF NOT EXISTS latlong (COUNTRY…
Alex
  • 11
  • 3
1
vote
1 answer

How to parse JSONs in Hive with Mixed Struct/String values for a single field?

I've got JSON data whose structure is similar to this: root |-- TX: struct (nullable = true) | |-- ARGS: array (nullable = true) | | |-- element: string (containsNull = true) | |-- COOKIE: array (nullable = true) | | |--…
nxl4
  • 714
  • 2
  • 8
  • 17
1
vote
1 answer

Map two columns into one on Athena using SerDe properties

I'm trying to map two columns into one on Athena using JsonSerDe properties. In this case, I want to map both columns conversionsRate and cr from jsons 1 and 2 to column cr_new (doing like a coalesce). json1 { "deviceType": "TABLET", "day":…
Lucas Mendes Mota Da Fonseca
  • 2,172
  • 2
  • 18
  • 28
1
vote
0 answers

How create a table with a serde where text file has fixed-width (non-delimited) columns?

I'm trying to create a table with a textfile without delimiter. row exemple: 1000000000168999337200----------030420191455594197981209954------- 00000240000005010000000011800000000000 CREATE EXTERNAL TABLE IF NOT EXISTS p_bi.stg_cob…
lboniotti
  • 41
  • 1
  • 6
1
vote
0 answers

spark databricks cluster noclassdeffounderror during insert

Experimenting with Databricks Spark cluster. When creating a table in a Hive database, I get the following error the first time. 19/06/18 21:34:17 ERROR SparkExecuteStatementOperation: Error running hive query:…
Anirban Kundu
  • 131
  • 1
  • 6
1
vote
1 answer

Hive import only certain file types from hdfs

I’m creating a external table using Serde org.apache.hive.hcatalog.data.JsonSerde Hdfs folder location has multiple file formats and I would like to import only json files types . I tried to use **SERDEPROPERTIES (“input.regex” = “*.json”)**…
RData
  • 959
  • 1
  • 13
  • 33
1
vote
1 answer

hive create table input.regex - filter out all rows starting with a char

I want to create table in Hive CREATE TABLE table ( a string ,b string ) PARTITIONED BY ( pr_filename string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ('input.regex'='reg_exp') ; but…
jmt
  • 719
  • 1
  • 9
  • 28
1
vote
1 answer

Can we create several entries from one line?

My logs look like this: client_id;event_1;event_2;event3 And i would like to get an SQL Table like this: client_id | event --------------------- ... | event_1 ... | event_2 ... | event_3 I am new to Hive, it seems to me…
Cinn
  • 4,281
  • 2
  • 20
  • 32
1
vote
1 answer

Creating table with Regex Serde in Hive returns error

I created a table using Regex Serde in Hive. In Hue it returns that the creation of the table was successful. However, when I try to return the table SELECT * FROM pricefile_edited or view the table in hue, it doesn't work and I get the Error . The…
QBits
  • 121
  • 1
  • 11
1
vote
1 answer

Hive SerDe returns error with JSON tweets Flume

I am collecting twitter stream data using Flume and storing it in JSON format in HDFS. I am trying to use Hive SerDe to put this twitter data into Hive table but I am getting a very frustrating error. hive> ADD JAR…
user10240542
1
vote
1 answer

Auto selected delimiter while using csv-serde

Have created a table using: ) ROW FORMAT SERDE 'com.bizo.hive.serde.csv.CSVSerde' with serdeproperties( "field.delim"="|" ) When I tried to upload a file, that is comma-delimited, it has been uploaded correctly. How is it possible, if I had…
Anastasia
  • 25
  • 1
  • 4
1
vote
3 answers

character slash is not being read by hive on using OpenCSVSerde

I have defined a table on top of files present in hdfs. I am using the OpenCSV Serde to read from the file. But, '\' slash characters in the data are getting omitted in the final result set. Is there a hive serde property that I am not using…
ForeverLearner
  • 1,901
  • 2
  • 28
  • 51
1
vote
1 answer

Using JSON Serde : java.net.URISyntaxException

I am new to Hive, Can anybody please help me with the below Error I receive when trying to create the following table: hive> create table Employees( > name String, > salary float, > subordinates array, > deductions…
Parijat Chakraborty
  • 37
  • 5
1
vote
1 answer

insert data into table using csv file in HIVE

CREATE TABLE `rk_test22`( `index` int, `country` string, `description` string, `designation` string, `points` int, `price` int, `province` string, `region_1` string, `region_2` string, `taster_name` string, `taster_twitter_handle` string,…
Harshit Mehta
  • 50
  • 7
1 2 3
…
10 11