0

I have a SingleOutputStreamOperator<ObjectA> objAStream and need to sink only the List<ObjectB> list objects which are in ObjectA stream in to kafka.

    public class ObjectA {
        public int id;
        public List<ObjectB> objBList;

        public ObjectA() {
        }
    }

SingleOutputStreamOperator<ObjectA> objAStream = someStream.map(new SomeMapper());

//required to sink only the member `ObjectB` objects in `objAStream` ObjectA

I'm new to flink and will it be possible to do it using FlapMap or any other way. Kindly give your comments.

chandana.ranasinghe
  • 1
  • 1

1 Answers1

0

I believe you need something along these lines:

DataStream<List<ObjectB>> objBStream = objAStream
  .map(a -> a.objBList)
  .returns(TypeInformation.of(new TypeHint<List<ObjectB>>(){}));

objBStream.addSink(...);

You might try this without the returns(...) clause, but I expect that will fail at runtime. The problem is that Flink needs to be able to serialize the data being streamed, and due to type erasure, the generic type info (ObjectB in this case) isn't available.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • Thanks David, yes with returns(...) it fails at runtime. Without returns it seems I need to implement the 'ResultTypeQueryable' interface as per the thrown exception. "could not be determined automatically, due to type erasure. You can give type information hints by using the returns(...) method on the result of the transformation call, or by letting your function implement the 'ResultTypeQueryable' interface." – chandana.ranasinghe Sep 13 '21 at 08:05
  • You can supply the necessary type info via the returns(...) method or via the ResultTypeQueryable interface. I've modified my answer to suggest `.returns(TypeInformation.of(Types.LIST(ObjectB.class)))`, but that might not be quite how it needs to be specified. It depends, in part, on whether ObjectB is a a proper POJO from Flink's pov. – David Anderson Sep 13 '21 at 18:56