Java 8 Streams | Average words for lines in textfile

Question

I'm trying to do some exercise on Streams and encountered following problem:

I have a text file, and I want to compute the average number of words per line. Can someone tell me if my way of thinking is correct? Here's some pseudocode I think should do the trick once implemented:

double wordCount(String filepath){
  return Files.lines(Paths.get(filepath))
                      // make a wordarray of the line
                      // average the size of every wordarray with something like that
                          (collect(Collectors.averagingDouble())

Can someone please help me with that?

How can I convert a line to a String array of words?

How do I get the size of that array?

Actually you mention several tasks. How to convert a line to an array of Strings. How to get the size of this array. Solve these sub tasks and bring your work to this question. — , May 13 '18 at 18:38
*"How do I get the size of that array?"* Really? `arr.length`!!! — Andreas, May 13 '18 at 18:54
@Andreas Yes I know that's arr. length... Somehow I thought to complicated and didn't think it's that easy with streams... — helloWorld, May 14 '18 at 06:16

azro · Accepted Answer · 2018-05-14T08:29:26.660

4

The 2 steps missing are :

How can I convert a line to a Stringarray of words : split around spaces >> s.split(" ");
How do I get the size of that array : get its length >> arr.length

Using specialized operation on IntStream

double wordCount(String filepath) {
    try {
        return Files.lines(Paths.get(filepath))
                .map(s -> s.split(" "))
                .mapToInt(arr -> arr.length)
                .average()
                .orElse(-1.0);
    } catch (IOException e) {
        e.printStackTrace();
    }
    return -1.0;
}

Less specific Collectors operation (To avoid againt prop 1.)

double wordCount2(String filepath) {
    try {
        return Files.lines(Paths.get(filepath))
                .map(s -> s.split(" "))
                .map(arr -> arr.length)
                .collect(Collectors.averagingDouble(Double::new));
    } catch (IOException e) {
        e.printStackTrace();
    }
    return -1.0;
}

edited May 14 '18 at 08:29

answered May 13 '18 at 18:48

azro

53,056
7
34
70

1

Nice, although I'd avoid using the second approach for this specific task. see https://stackoverflow.com/questions/49965753/when-to-use-specialised-stream-methods-over-methods-in-the-collectors-class-or-v – Ousmane D. May 13 '18 at 20:10
@azro Thank you very much! – helloWorld May 14 '18 at 06:18
1

@Aominè I’d avoid using `String.split` at all, when all I want, is the number of words/matches… – Holger May 14 '18 at 09:26

score 3 · Answer 2 · answered May 14 '18 at 09:22

You are doing unnecessary work by splitting a string into a bunch of strings, one for each word, populating an array with them, just to ask for the array’s size afterwards.

If you want to get the number of words, consider a straight-forward method, only counting the words

private static final Pattern WORD = Pattern.compile("\\w+");
public static int wordCount(String s) {
    int count = 0;
    for(Matcher m = WORD.matcher(s); m.find(); ) count++;
    return count;
}

You may then use this method inside a Stream operation, to get the average word count:

Files.lines(Paths.get(filePath)).mapToInt(YourClass::wordCount).average().orElse(0)

With Java 9, you could rewrite the wordCount method to use a Stream like

private static final Pattern WORD = Pattern.compile("\\w+");
public static int wordCount(String s) {
    return (int)WORD.matcher(s).results().count();
}

But the loop likely is more efficient as it omits the construction of MatchResult instances.

Nice! , looks compact and elegant. – Ousmane D. May 14 '18 at 09:33 — Ousmane D., May 14 '18 at 09:33
@Aominè especially the java-9, a true job for the scanner – Eugene May 14 '18 at 09:35 — Eugene, May 14 '18 at 09:35

score 1 · Answer 3 · answered May 14 '18 at 05:10

private static void wordcount(String filePath) throws IOException {
    Path path = Paths.get(Paths.get(filePath).toUri());
    double result = Files.lines(path).map(s -> {
        String trimmed = s.trim();
        if (trimmed.isEmpty()) return 0;
        else return trimmed.split(" ").length;
    }).mapToInt(i -> i.intValue()).average().getAsDouble();

    System.out.println(result);
}

Java 8 Streams | Average words for lines in textfile

3 Answers3