One-to-Many Processor streaming with JPA

Question

I have a SCDF processor which produces multiple outputs per input. The list of outputs could be rather large. That's why I was thinking of streaming the results from JPA repository.

@Service
public class ProcessorFunction implements Function<Parent, Stream<Message<Child>>> {

    @Autowired
    private ChildRepository repo;

    @Override
    @Transactional(readOnly = true)
    public Stream<Message<Child>> apply(Parent parent) {
        try (var stream = repo.findByParentId(parent.getId())) {
            return stream.map(GenericMessage::new);
        }
    }
}

public interface ChildRepository extends  JpaRepository<Child, Long> {
  
  Stream<Child> FindByParentId(long id);
}

I expected that the stream will send the list of output on the output channel. However, only null values are being sent.

I have gone through spring-cloud-dataflow document as well as spring-cloud-stream documentation but could not find my use case.

Kindly help!

You can't return a stream of persistent objects out of a transaction — Maurice Perry, Nov 04 '22 at 10:57
! Well that's a bummer. What else can I do to prevent all the results loading in memory? Can I convert them into a DTO and send? Alternatively, can I send them to output channel programmatically from within a transaction? — Juzer Ali, Nov 04 '22 at 11:06
I don't know what you're planning to do with the results, but you have basically two options: 1) do whatever you have to do during the transaction 2) save the results in a local intermediate storage (file) — Maurice Perry, Nov 04 '22 at 11:57
I suggest exploring the JPA finder method returning a Flux and then you also return a Flux from the apply method by mapping from result. — Corneil du Plessis, Dec 09 '22 at 15:11

One-to-Many Processor streaming with JPA

0 Answers0