Flume Hive sink failed to serialize JSON with array

Question

I am trying to load JSON data to Hive via Hive Sink. But it fails with the following error:

WARN org.apache.hive.hcatalog.data.JsonSerDe: Error [java.io.IOException: Field name expected] parsing json text [{"id": "12345", "url": "https://mysite", "title": ["MyTytle"]}].
INFO org.apache.flume.sink.hive.HiveWriter: Parse failed : Unable to convert byte[] record into Object  : {"id": "12345", "url": "https://mysite", "title": ["MyTytle"]}

Example of data:

{"id": "12345", "url": "https://mysite", "title": ["MyTytle"]}

Description of Hive table:

id              string                                      
url             string                                      
title           array<string>                               
time            string                                      

# Partitions
time            string

And the same way it works fine if JSON data doesn't contain arrays (and Hive table either).

Flume version: 1.7.0 (Cloudera CDH 5.10)

Does it possible to load JSON data with arrays via Flume Hive sink?

score 0 · Answer 1 · edited May 23 '17 at 12:16

Is it possible to load JSON data with arrays via Flume Hive sink?

I assume it is possible, despite I never tried myself. From:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_HDP_RelNotes/content/ch01s08s02.html

Following serializers are provided for Hive sink:

JSON: Handles UTF8 encoded Json (strict syntax) events and requires no configuration. Object names in the JSON are mapped directly to columns with the same name in the Hive table. Internally uses org.apache.hive.hcatalog.data.JsonSerDe but is independent of the Serde of the Hive table. This serializer requires HCatalog to be installed.

So maybe you are implementing something wrong in the SerDe. This user solved the problem of serialising a JSON with arrays by performing a previous regexp:

Parse json arrays using HIVE

Another thing you may try is to change the SerDe. At least you have this two options (maybe there are some more):

'org.apache.hive.hcatalog.data.JsonSerDe'
'org.openx.data.jsonserde.JsonSerDe' (https://github.com/sheetaldolas/Hive-JSON-Serde/tree/master)

Flume Hive sink failed to serialize JSON with array

1 Answers1