Stack Overflow
Stack Overflow

Questions tagged [hive-serde]

SerDe is short for Serializer/Deserializer, an interface used by Hive for both serialization and deserialization during IO and also interpreting the results of serialization as individual fields. A SerDe allows Hive to read in data from a table, and write it back out to HDFS in any custom format. Anyone can write their own SerDe for their own data formats.

Official documentation page: SerDe

There are many SerDe bundled with Hive as well as third-party SerDe, such as:

  • LazySimpleSerDe
  • OpenCSVSerDe
  • RegexSerDe
  • JsonSerDe
  • AvroSerDe
  • ParquetHiveSerDe
  • OrcSerDe
  • MultiDelimitSerDe
164 questions
1
vote
1 answer

Can I use 2 fields terminators(like ',' and '.') at a time in hive while creating table?

I have a file with id and year. My fields are separated by , and .. Is there any chance I can in the place of fields terminated by can I use , and .?
user2815076
  • 29
  • 7
1
vote
1 answer

json file to hive external table

I have data in json…
Kobra
  • 313
  • 1
  • 15
1
vote
0 answers

Fetch query params from S3 access log using Athena

I wish to fetch a map of query params from S3 access log using Athena. E.g. for the following log line example: 283e.. foo [17/Jun/2017:23:00:49 +0000] 76.117.221.205 - 1D0.. REST.GET.OBJECT 1x1.gif "GET…
daddycool
  • 11
  • 4
1
vote
1 answer

Create hive table from JSON data

I have a file with Json data which takes the below form: Ex: { "Name": "xxxx", "Address": [{ "Street": "aa", "City": "bbb" }, { "Street": "ccc", "City": "ddd", "Country": "eee" }] } The…
Cheater
  • 435
  • 1
  • 4
  • 16
1
vote
1 answer

XML Schema to Hive Schema

I am trying to load a xml file into hive table. I am using xml serde here. I am able to load simple flat xml files. But when there are nested elements in the xml, I am using hive complex data types to store them (for e.g., array). Below is…
Sri Harsha Chennavajjala
  • 179
  • 2
  • 17
1
vote
1 answer

Remove/Mapping duplicates key on Hive table?

I have JSON files to be loaded to hive table, but it contains duplicate key that make all the data null or unable to be select queried on Hive. Those JSON file had something like this…
code-addict
  • 137
  • 2
  • 12
1
vote
1 answer

Loading nested XML data into Hive using SerDe

I am trying load nested XML data into Hive. Sample data is as follows... ALFKI 10643 ALFKI …
Sri Harsha Chennavajjala
  • 179
  • 2
  • 17
1
vote
1 answer

Loading JSON data into hive tables

I've tried loading simple JSON records from a file into hive tables like as shown below. Each JSON record is in a separate…
Sri Harsha Chennavajjala
  • 179
  • 2
  • 17
1
vote
0 answers

Hive error: cannot validate serde: org.apache.hadoop.serde2.RegexSerde

I am very new here, I am trying to run the following code on my cloudera quickstart VM. CREATE TABLE apache_common_log ( host STRING, identity STRING, user STRING, time STRING, request STRING, status STRING, size STRING ) ROW FORMAT…
sparkmeow
  • 21
  • 5
1
vote
2 answers

java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe not found in Hive

I am trying to process multicharacter delmiter in hive. I already created a table with the same successfully create external table showtmp3(doc_name STRING,doc_content STRING) row format SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' WITH…
USB
  • 6,019
  • 15
  • 62
  • 93
1
vote
1 answer

XML Hive Serde Extract timestamp Hadoop

I am trying to extract the timestamp from an xml using xml serde in Hive. The external table creation links to a hdfs directory. Currently, the timestamp value shows up as null in my table. I am thinking the timestamp needs to be casted? I am not…
Defcon
  • 807
  • 3
  • 15
  • 36
1
vote
1 answer

How to integrate the java code written on local machine to Serde for hive query

I have written code (java) for the conversion of EBCDIC to ASCII on the local machine. And we are writing a serde for hive query which would provide the ability for Hive to read the EBCDIC file and convert it into ASCII file at run time. So…
tejas bharambe
  • 15
  • 4
1
vote
4 answers

Loading JSON file in HIVE table

I have a JSON file like below, which I want to load in a HIVE table with parsed format, what are possible options I can go for. If it is AVRO then I could have used directly AvroSerDe. But the source file in this case is JSON. { …
Koushik Chandra
  • 1,565
  • 12
  • 37
  • 73
1
vote
2 answers

Hive external table with JSON SerDe fetching all NULL values

My data is stored in HDFS at directory /tmp/kafka/alert in multiple files. Each file contain new-line separated JSON objects like following. {"alertHistoryId":123456,"entityId":123,"deviceId":"123","alertTypeId":1,"AlertStartDate":"Dec 28, 2016…
Arjit
  • 421
  • 7
  • 20
1
vote
1 answer

Trying to create an external table from JSON in hive

{"name":"Our Example","link_params":{"cid":"eml_mmj_{date('yyyyMMdd')}"},"template":"Great Example Television","day":"2016-10-02","list_name":"Example Television","id":5112652} Here's an example row from the json file. CREATE TABLE…
firestreak
  • 377
  • 4
  • 17
1 2 3
…
10 11