2

I have an avro input_stream something like this:

VERSION STRING,
ID STRUCT<
  NAME STRING,
  STATUS STRING
>,
ROLES ARRAY<
  STRUCT<
    DESC STRING,
    TYPE STRING
  >
>

and I want to falatten it and do some filtering as well.

This is working:

select version, explode(roles)->type from input_stream where version='1.0' emit changes;

but this fails:

select version, explode(roles)->type from input_stream where explode(roles)->type='master' emit changes;

Invalid Predicate: Can't find any functions with the name 'EXPLODE'. expression:(explode(roles)->type='master')

How can I filter directly the array elements based on type without creating a new stream that will query and filter from this exploded one?

Thanks!

Example how to reproduce:

>create stream test_for_robin_moffatt (VERSION STRING,ID STRUCT<NAME STRING,STATUS STRING>,ROLES ARRAY<STRUCT<DESC STRING,TYPE STRING>>) with (kafka_topic='test_for_robin_moffatt', value_format='json', partitions=1);

    
>insert into TEST_FOR_ROBIN_MOFFATT (VERSION,ID, ROLES) values ('1.1', STRUCT(name:='zoltan', status:='active'), ARRAY[STRUCT(desc:='blabla', type:='master')]);
    
>SET 'auto.offset.reset' = 'earliest';
    
>select version, id->name, id->status, explode(roles)->desc, explode(roles)->type from TEST_FOR_ROBIN_MOFFATT emit changes;
>select version, id->name, id->status, explode(roles)->desc, explode(roles)->type from TEST_FOR_ROBIN_MOFFATT where version='1.1' emit changes;
>select version, id->name, id->status, explode(roles)->desc, explode(roles)->type from TEST_FOR_ROBIN_MOFFATT where id->status='active' emit changes;
    
The statements above work fine:
        
+---------------------------------------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+
|VERSION                                      |NAME                                         |STATUS                                       |DESC                                         |TYPE                                         |
+---------------------------------------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+---------------------------------------------+
|1.1                                          |zoltan                                       |active                                       |blabla                                       |master                                       |

But, this fails:        
        
>select version, id->name, id->status, explode(roles)->desc, explode(roles)->type from TEST_FOR_ROBIN_MOFFATT where explode(roles)->type='master' emit changes;
Invalid Predicate: Can't find any functions with the name 'EXPLODE'. expression:(EXPLODE(ROLES)->TYPE = 'master'), schema:`VERSION` STRING, `ID` STRUCT<`NAME` STRING, `STATUS` STRING>, `ROLES` ARRAY<STRUCT<`DESC` STRING, `TYPE` STRING>>, `ROWTIME` BIGINT
user213546
  • 341
  • 1
  • 7

0 Answers0