Stack Overflow
Stack Overflow

Questions tagged [hive-serde]

SerDe is short for Serializer/Deserializer, an interface used by Hive for both serialization and deserialization during IO and also interpreting the results of serialization as individual fields. A SerDe allows Hive to read in data from a table, and write it back out to HDFS in any custom format. Anyone can write their own SerDe for their own data formats.

Official documentation page: SerDe

There are many SerDe bundled with Hive as well as third-party SerDe, such as:

  • LazySimpleSerDe
  • OpenCSVSerDe
  • RegexSerDe
  • JsonSerDe
  • AvroSerDe
  • ParquetHiveSerDe
  • OrcSerDe
  • MultiDelimitSerDe
164 questions
0
votes
1 answer

Hive - Replace columns in ORC table

I have a hive table saved in ORC files, this is the definition in the "create" command: ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT …
Bramat
  • 979
  • 4
  • 24
  • 40
0
votes
1 answer

Hive XML Serde - Parsing strings with line breaks

I'm using hivexmlserde-1.0.5.3 to parse XML data into Hive tables. I'm facing an issue when I'm trying to parse tags which have line breaks in them, something like this : 1 …
kndarp
  • 101
  • 1
  • 3
  • 9
0
votes
1 answer

Hive error while creating table in ORC format

I am getting error while creating table in hive in ORC format. Below is the create table script. CREATE TABLE tgt_ebr_agreements_item_notes STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT…
Pranav
  • 363
  • 8
  • 19
0
votes
1 answer

Hive-XML-SerDe - Key/Value Pair - Map

USA West
Number 135 Street …
Paciferous
  • 21
  • 2
0
votes
2 answers

Hive table delimited by comma and multiple spaces

I have a similiar question to here: Hive table source delimited by multiple spaces My data looks like this: AL, 01, 2016010700, , BEST, 0, 266N, 753W AL, 01, 2016010706, , BEST, 0, 276N, 747W AL, 01, 2016010712, , BEST, 0, 287N, …
BlueElephant
  • 1
  • 1
  • 2
0
votes
1 answer

HIVE XML SerDe: Failed with exception java.io.IOException:java.lang.NullPointerException

I create an external table with HIVE (Hive 2.1.1-mapr-1703) from an XML file with XML SerDe. The file is the XML example from the W3C consortium. This is my code to create the table: add jar /mapr/localpath/hivexmlserde-1.0.5.3.jar; USE…
Frank
  • 1,315
  • 1
  • 10
  • 14
0
votes
1 answer

java.lang.ClassCastException: org.apache.hadoop.hive.ql.io.orc.OrcStruct cannot be cast to org.apache.hadoop.io.Text. Error with json serde

I am new to working with json data on hive. I am working on a spark application that gets json data and stores it into hive tables. I have a json like this: which looks like this when expanded: I am able to read the json into a dataframe and save…
Hemanth Annavarapu
  • 823
  • 3
  • 19
  • 37
0
votes
1 answer

Hive - XML- Serde - Key Value Pair - Create table

Key Value pair combination. We are trying to explode the ID as a column name and VALUE as the corresponding data for each column. ` Column1 25672 Column2…
Paciferous
  • 21
  • 2
0
votes
0 answers

Hive XML SerDe - Add a constant value in column

I have a Hive table like below..I would like have the first column to be a constant value (Its not coming in the XML file). if I hardcode the value, its not being displayed when I do a select on the table... what am I doing wrong? I tried using the…
vitruvian
  • 63
  • 1
  • 5
0
votes
0 answers

Complex XML schema to Hive schema

I am trying to load a xml file into hive table. I am using xml serde [here][1]. I am able to load simple flat xml files. But when there are nested elements in the xml, I am using hive complex data types to store them (for e.g., array). Below…
Sri Harsha Chennavajjala
  • 179
  • 2
  • 17
0
votes
1 answer

Prevent Inserting NULL while using Hive Regex Serde

RegexSerDe uses regular expression (regex) to deserialize data. It doesn't support data serialization. It can deserialize the data using regex and extracts groups as columns. In deserialization stage, if a row does not match the regex, then all…
Viswaram Ranganathan
  • 11
  • 1
0
votes
2 answers

Hive from JSON Error

I can't make this json into hive table somehow, either become all null data or not able being selected. i just need all the same fields with my DDL, and if it's structured inside it, i want to let it as a string instead try to parse that. The only…
code-addict
  • 137
  • 2
  • 12
0
votes
1 answer

regex for specific delimiter string in Hive serde

I use serde to read data with specific format with delimiter | One line of my data may looks like: key1=value2|key2=value2|key3="va , lues", and I create the hive table as below: CREATE EXTERNAL TABLE( field1 STRING, field2 STRING, field3…
rmnvnv
  • 1
0
votes
2 answers

How do you add Data to an Existing Hive Metastore?

I have multiple subdirectories in S3 that contain .orc files. I'm trying to create a hive metastore so I can query the data with Presto / Hive, etc. The data is poorlly structured (no consistent delimiter, ugly characters, etc). Here's a scrubbed…
TheProletariat
  • 916
  • 2
  • 11
  • 23
0
votes
1 answer

Hive Custom Serde

I am new to Hive, got some stuff to parse logs of the format [Time Stamp] {Complex JSON data} As I see from my searches so far, There are JSON Serde's available. Can I extend those JSON Serde code to suit my need ? If so which JSON serde code…
veera
  • 69
  • 8
1 2 3
…
10
11