Stack Overflow
Stack Overflow

Questions tagged [hive-serde]

SerDe is short for Serializer/Deserializer, an interface used by Hive for both serialization and deserialization during IO and also interpreting the results of serialization as individual fields. A SerDe allows Hive to read in data from a table, and write it back out to HDFS in any custom format. Anyone can write their own SerDe for their own data formats.

Official documentation page: SerDe

There are many SerDe bundled with Hive as well as third-party SerDe, such as:

  • LazySimpleSerDe
  • OpenCSVSerDe
  • RegexSerDe
  • JsonSerDe
  • AvroSerDe
  • ParquetHiveSerDe
  • OrcSerDe
  • MultiDelimitSerDe
164 questions
0
votes
2 answers

How to parse nested Json structure in Hive?

I have json like the following in hadfs. {"result": [{"sys_tags": {"display_value": "d1", "value": "v1"}, "user_input": {"display_value": "d2", "value": "v2"}}, {"sys_tags": {"display_value": "d1", "value": "v1"}, "user_input": {"display_value":…
Manish Kumar
  • 245
  • 1
  • 3
  • 7
0
votes
0 answers

Unable to run SerDe

We have one ebcdic sample file. It is stored in /user/hive/warehouse/ebcdic_test_file.txt Cobol layout of the file is stored in /user/hive/Warehouse/CobolSerde.cob We are running on Hue browser query editor. We also tried in CLI. But the same…
tejas bharambe
  • 15
  • 4
0
votes
1 answer

unable to get avro data for array> in rows

Hi I have a avro schema with array of struct and I am able to save data as avro. but while retrieving the data from array> I am unable to get in rows. All the data I am getting in single row. here is the table definition…
AJ007
  • 125
  • 2
  • 11
0
votes
1 answer

Error while trying to load json file into external table in hive

Data looks like: {"custId":1185972,"movieId":null,"genreId":null,"time":"2012-07-01:00:00:07","recommended":null,"activity":8} The query i am running is: add jar /home/student/hive-0.11.0-bin/lib/json-serde-1.3.7-jar-with-dependencies.jar; CREATE…
Andy Reddy
  • 93
  • 2
  • 9
0
votes
1 answer

Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.apache.hadoop.hive.serde2.avro.AvroSerde

I am creating a hive table using avro serde to store twitter data. Here's the code- create table tweets row format serde 'org.apache.hadoop.hive.serde2.avro.AvroSerde' stored as inputformat …
vinay
  • 167
  • 1
  • 4
  • 12
0
votes
1 answer

Trying to change the cloumn type in hive, running a DDL, but it's throwing an error "

Trying to change the cloumn type in hive, running a DDL, but it's throwing an error Running the following DDL: ALTER TABLE INV.HTL_RATE_PLAN CHANGE RATE_PLAN_RSTRCT_STRT_DT RATE_PLAN_RSTRCT_STRT_DT DATE COMMENT 'Advance booking alternate days…
ruthvik
  • 11
  • 1
0
votes
1 answer

Hive table source delimited by multiple spaces

How can I make the following table source delimiter by one or more white spaces: CREATE EXTERNAL TABLE weather (USAF INT, WBAN INT, `Date` STRING, DIR STRING, SPD INT, GUS INT, CLG INT, SKC STRING, L STRING, M STRING, H STRING, VSB DECIMAL, MW1…
abutmah
  • 63
  • 1
  • 3
  • 9
0
votes
1 answer

Unable to load hive table with fixed width file using SerDe

I have fixed width file having 5 fixed width column. Here is the structure of the file. {col1:3char, col2:35char, col3:3char, col4:11char, col5:4char}. Here is the sample file 111 SagarKhatavkar 030 9999ABIT 112 …
Sagar
  • 43
  • 1
  • 6
0
votes
1 answer

Deserialize protobuf column with Hive

I am really new to Hive, I apologize if there are any misconceptions in my question. I need to read a hadoop Sequence File into a Hive table, the sequence file is thrift binary data, which could be deserialized using SerDe2 that comes with Hive.…
emiaozang
  • 1
  • 1
0
votes
1 answer

Data (Single quotes and Doube Quotes) Mismatch in Hive

While loading the file from mainframe into Hadoop in ORC format,some of the data loaded with Single Quotes(') and remaining with Double quotes(").But the complete source file is in Single Quote ('). To specify custom delimiters used Hive Cobol…
Revathi
  • 31
  • 2
  • 6
0
votes
2 answers

Converting EBCDIC to ASCII file format- SerDe for hive. How to overcome issue of delimiter in EBCDIC file

There is nothing like /n in ebcdic. There is no support for new line in ebcdic. How should I convert that like. There is no delimiter in ebcdic. So while converting this file. How shall I know that new line has come? Suggestions please.
tejas bharambe
  • 15
  • 4
-1
votes
1 answer

taking text file into hive database

i trying to run this code for so long time can somebody tell me what is problem in it code :- CREATE EXTERNAL TABLE samp_log ( ip String ,col1 String ,col2 String , date String , time_hour int ,time_min int ,time_sec int ,zone int , request…
Avinash Jadhav
  • 53
  • 1
  • 1
  • 6
-2
votes
2 answers

Error in Nested JSON in HIve

I was trying to load this json data in hive { "id": "0001", "type": "donut", "name": "Cake", "ppu": 0.55, "batters": { "batter": [ { "id": "1001", "type": "Regular"…
unchained
  • 19
  • 6
-3
votes
1 answer

Hive RegEx Serde doesn't work and loads as NULL

I am trying to create a hive table with Serde Regex property to load the below file. Input File: $ hdfs dfs -cat /user/t04413b/test.log {"repoType":3,"repo":"PROD_hive","reqUser":"shdingst","evtTime":"2020-06-09 01:01:23.308"} Hive create table…
Ragul Cs
  • 11
  • 1
1 2 3
…
10
11