Can I split a stream into multiple smaller streams

Question

There are mulitple questions for streams but for this usecase & in java, didnt find any.

I have a huge stream of objects Stream<A> [~1Million objects]. StreamA comes from a file.

Class A { enum status [Running,queued,Completed], String name }

I want to split Stream<A> into three streams without using any Collect statements. Collect statement loads everything into memory.

I am facing StackOverflowException as I am calling stream.concat multiple times here.

Stream.Concat has problem mentioned in Java Docs "Implementation Note: Use caution when constructing streams from repeated concatenation. Accessing an element of a deeply concatenated stream can result in deep call chains, or even StackOverflowException."

Map<Status, Stream<String>> splitStream = new HashMap<>();
streamA.foreach(aObj -> 
Stream<String> statusBasedStream = splitStream.getOrDefault(aObj.status,Stream.of());
splitStream.put(aObj.status, Stream.concat(statusBasedStream, Stream.of(aObj.name)));

There are few options where custom streams are available in github to achieve Concatenation but wanted to use standard libraries to solve this.

If data is smaller would have taken a list approach as mentioned here (Split stream into substreams with N elements)

"Collect statement loads everything into memory." Sure, but putting everything into a map also loads everything into memory. There's no laziness there. — Andy Turner, May 05 '20 at 00:38
correct but anything which loads into memory, I would get OutOfMemory exception. — Vishwaram Sankaran, May 05 '20 at 00:48
This feels like an XY problem. What are you really trying to achieve? — David Conrad, May 05 '20 at 01:17
What are you going to do with the almost 1 million names that are status `Completed`, if not load them into memory? — Andreas, May 05 '20 at 01:19
I want to write them into file. And FileWriting is factored out into a seperate class. Didnt want to have open file Handlers for diff status(We have almost 6 status). I also want to Gzip the files. — Vishwaram Sankaran, May 05 '20 at 01:25
Just now came across Stream.Builder builderS = Stream.builder(); I am going to try this. When i did Stream.concat above, since it is lazy loaded, got into StackOverflow. If i load everything into memory getting OOM. I can do file writing, just that my code modularity goes for a toss. — Vishwaram Sankaran, May 05 '20 at 01:26
@VishwaramSankaran What is your issue with having 6 open files? That is nothing, and it is what you need to do for this. — Andreas, May 05 '20 at 01:50
Agreed it also works. Just that i had factored out file handling into. a separate class. It looks more modular if i have this stream split earlier. I have taken a Function rowMapper, need to pass this function also to the file Handler class. I have used this **StreamBuilder**, it had achieved whatever i had intended for. Thanks @Andreas for taking your time. — Vishwaram Sankaran, May 05 '20 at 03:17
A million strings are not that impressive. The fact that you didn’t even realize that all strings are in memory when you collect them into `Stream.Builder` instances, indicates that there is no actual heap memory problem with them. You didn’t say where this `Stream` does come from. Further, it’s not clear what problems you get with `Stream.concat`, when you talk about splitting, the very opposite of what `concat` does. — Holger, May 05 '20 at 08:55
Ok taking the solution which andreas proposed with having mulitple file handlers open. — Vishwaram Sankaran, May 05 '20 at 20:16
Why are you calling `concat` multiple times? It seems to be entirely unrelated to the other problem description. Are you processing multiple input files? Stream over the files and use `flatMap`, to create a single stream over the contained elements. — Holger, May 06 '20 at 09:35

score 1 · Answer 1 · answered May 05 '20 at 11:27

Not the exact solution of the problem but if you have information about the indexes then combination of Stream.skip() and Stream.limit() can help in this - Below is the dummy code that I tried -

    int queuedNumbers = 100;
    int runningNumbers=200;
    Stream<Object> all = Stream.of();
    Stream<Object> queuedAndCompleted = all.skip(queuedNumbers);
    Stream<Object> queued = all.limit(queuedNumbers);
    Stream<Object> running = queuedAndCompleted.limit(runningNumbers);
    Stream<Object> completed = queuedAndCompleted.skip(runningNumbers);

Hope it would be of some help.

Can I split a stream into multiple smaller streams

1 Answers1