Hive with python transform function: "cannot recognize input near 'transform'" error

Question

I have a Hive table that tracks the status of an object moving through stages of a process. The table looks like this:

hive> desc journeys;
object_id           string                                      
journey_statuses    array<string>

Here's a typical example of a record:

12345678    ["A","A","A","B","B","B","C","C","C","C","D"]

The records in the table were generated using Hive 0.13's collect_list and the statuses have an order (if order wasn't important, I would have used collect_set). For each object_id, I'd like to abbreviate the journey to return the journey statuses in the order that they appear.

I wrote a quick Python script that reads from stdin:

#!/usr/bin/env python
import sys
import itertools

for line in sys.stdin:
    inputList = eval(line.strip())
    readahead = iter(inputList)
    next(readahead)
    result = []
    for id, (a, b) in enumerate(itertools.izip(inputList, readahead)):
        if id == 0:
          result.append(a)
        if a != b:
          result.append(b)
    print result

I planned to use this in a Hive transform call. It seems to work when run locally:

$ echo '["A","A","A","B","B","B","C","C","C","C","D"]' | python abbreviate_list.py
['A', 'B', 'C', 'D']

However, when I add the file and try to execute within Hive, an error is returned:

hive> add file abbreviateList.py;                                                                           
Added resource: abbreviateList.py

hive> select
    >   object_id,
    >   transform(journey_statuses) using 'python abbreviateList.py' as journey_statuses_abbreviated
    > from journeys;
NoViableAltException( ... wall of Java error messages ... )
FAILED: ParseException line 3:2 cannot recognize input near 'transform' '(' 'journey_statuses' in select expression

Can you see what I'm doing wrong?

score 5 · Accepted Answer · edited May 23 '17 at 12:15

Apparently you can't SELECT other fields that are not in the transform (in your example, object_id). This other SO question seems to indirectly address that:

How can select a column and do a TRANSFORM in Hive?

You could in theory modify your Python to accept the object_id as an input parameter and make it a passthrough to another output field if you need it to be included in the output.

Hive with python transform function: "cannot recognize input near 'transform'" error

1 Answers1