How to chain multiple different InputStreams into one InputStream

Question

I'm wondering if there is any ideomatic way to chain multiple InputStreams into one continual InputStream in Java (or Scala).

What I need it for is to parse flat files that I load over the network from an FTP-Server. What I want to do is to take file[1..N], open up streams and then combine them into one stream. So when file1 comes to an end, I want to start reading from file2 and so on, until I reach the end of fileN.

I need to read these files in a specific order, data comes from a legacy system that produces files in barches so data in one depends on data in another file, but I would like to handle them as one continual stream to simplify my domain logic interface.

I searched around and found PipedInputStream, but I'm not positive that is what I need. An example would be helpful.

score 111 · Accepted Answer · edited May 23 '17 at 11:54

111

It's right there in JDK! Quoting JavaDoc of SequenceInputStream:

A SequenceInputStream represents the logical concatenation of other input streams. It starts out with an ordered collection of input streams and reads from the first one until end of file is reached, whereupon it reads from the second one, and so on, until end of file is reached on the last of the contained input streams.

You want to concatenate arbitrary number of InputStreams while SequenceInputStream accepts only two. But since SequenceInputStream is also an InputStream you can apply it recursively (nest them):

new SequenceInputStream(
    new SequenceInputStream(
        new SequenceInputStream(file1, file2),
        file3
    ),
    file4
);

...you get the idea.

3

Doh! I must have really bad luck with my keywords when googling today. Because I gave it a good googling. I figured that I can't really be the first one needing this, thanks! – Magnus Jan 12 '13 at 16:54
5

SequenceInputStream constructor also accepts a Enumeration, so you don't have to do the nesting. – adi Oct 17 '13 at 18:44
6

Are they any concerns if I chain hundreds of inputstreams using this technique? – pretzels1337 Dec 04 '13 at 18:06
1

@pretzels1337, a bit late. yes there is a concern : you will open all files at once, when creating the enumeration of inputstream, and there is a limit to the number of files that can be opened concurrently (set by the OS). For this use case, I would rather copy the SequenceInputStream code, replace the `Enumeration` with `List`, and alter the `nextStream()` and `close()` methods accordingly. You could provide your own `Enumeration` that opens the stream in `nextElement()`, but look at what will happen in `close()` in case of IOExcpetion on the first inputstream – Thierry Nov 15 '16 at 08:49
Can I seek to random position in SequenceInputStream? I mean forward and backward. – ed22 Feb 13 '18 at 12:49

score 16 · Answer 2 · answered Jan 12 '13 at 16:34

This is done using SequencedInputStream, which is straightforward in Java, as Tomasz Nurkiewicz's answer shows. I had to do this repeatedly in a project recently, so I added some Scala-y goodness via the "pimp my library" pattern.

object StreamUtils {
  implicit def toRichInputStream(str: InputStream) = new RichInputStream(str)

  class RichInputStream(str: InputStream) {
// a bunch of other handy Stream functionality, deleted

    def ++(str2: InputStream): InputStream = new SequenceInputStream(str, str2)
  }
}

With that, I can do stream sequencing as follows

val mergedStream = stream1++stream2++stream3

or even

val streamList = //some arbitrary-length list of streams, non-empty
val mergedStream = streamList.reduceLeft(_++_)

score 16 · Answer 3 · answered Jul 15 '16 at 15:13

Another solution: first create a list of input stream and then create the sequence of input streams:

List<InputStream> iss = Files.list(Paths.get("/your/path"))
        .filter(Files::isRegularFile)
        .map(f -> {
            try {
                return new FileInputStream(f.toString());
            } catch (Exception e) {
                throw new RuntimeException(e);
            }
        }).collect(Collectors.toList());

new SequenceInputStream(Collections.enumeration(iss)))

score 1 · Answer 4 · answered Sep 24 '13 at 10:04

Here is a more elegant solution using Vector, this is for Android specifically but use vector for any Java

    AssetManager am = getAssets();
    Vector v = new Vector(Constant.PAGES);
    for (int i =  0; i < Constant.PAGES; i++) {
        String fileName = "file" + i + ".txt";
         InputStream is = am.open(fileName);
         v.add(is);
    }
    Enumeration e = v.elements();
    SequenceInputStream sis = new SequenceInputStream(e);
    InputStreamReader isr = new InputStreamReader(sis);

    Scanner scanner = new Scanner(isr);   // or use bufferedReader

score 0 · Answer 5 · answered Jun 13 '18 at 09:30

Here's a simple Scala version that concatenates an Iterator[InputStream]:

import java.io.{InputStream, SequenceInputStream}
import scala.collection.JavaConverters._

def concatInputStreams(streams: Iterator[InputStream]): InputStream =
  new SequenceInputStream(streams.asJavaEnumeration)

How to chain multiple different InputStreams into one InputStream

5 Answers5

See also

Linked

Related