Adding Upon a Value in a Linkedhashmap

Question

I am iterating through several text files and I am trying to find the top 20 words amongst all the text files. I have managed to set up some code to find the top 20 words in a single file. However, now I am struggling with several files.

I have a global linked-hashmap where I want to store every new word (as a key) I come across in a text file and I want to update its value (the number of times it occurs) as I come across more of the word. For example in the first file, I find 8000 instances of the word "the" and in the next file I come across 7000 instances of "the" in another file then I want the value of the key "the" to be updated to 15000.

Here is my code:

import java.util.*;
import java.util.stream.Collectors;
import java.io.IOException;
import java.nio.file.*;
import java.util.Map.Entry;
import java.util.function.Function;
import java.io.File;
import java.nio.charset.StandardCharsets;

public class FileReaderTwo
{
    static LinkedHashMap<String, Long> top20Words = null;
    public static void main(String args[])
    {
        File dir = new File("data/");
        for (File file : dir.listFiles()) 
        {
            try
            {
                top20Words = Files.lines(Paths.get(file.toString()), StandardCharsets.ISO_8859_1)
                        .flatMap(line -> Arrays.stream(line.toLowerCase().split("[\\(,\\).\\s+]+")))
                        .collect(Collectors.groupingBy(Function.identity(), Collectors.counting())).entrySet().stream()
                        .sorted(Entry.comparingByValue(Comparator.reverseOrder()))
                        .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
                        .collect(Collectors.toMap(Entry::getKey, Entry::getValue, (u, v) -> u, LinkedHashMap::new));
            } catch (IOException e)
            {
                e.printStackTrace();
            }
        }
        System.out.println(top20Words);
    }
}

Note: I know that at the moment it prints out every word, I wanted to deal with this issue first and fix that later.

Answer 1

Okay, I modified this to do what I believe you are looking for. It works as follows.

• created two methods to facilitate handling exceptions for file access. Not only do I find this easier and cleaner it is the recommended approach as opposed to trying to position try class constructs within a stream.
• takes all the words and does a frequency count and stores them in a map.
• sorts the entrySet of the map and places the top 20 words (highes wordcount) in the map in descending order.

The overall result is to count all the words in multiple files and present the tally in descending order.

import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Arrays;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class FileWordCount {
    
    public static void main(String[] args) {
        FileWordCount fwc = new FileWordCount();
        Map<String,Long> map = fwc.getTheWords();
        map.entrySet().forEach(System.out::println);
    }
    
    // helper methods to handle exceptions.
    private  Stream<Path> getFiles(String dir) {
        try {
            return Files.list(Path.of(dir));
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }
    
    private  Stream<String> getLines(Path path) {
        try {
            return Files.lines(path,StandardCharsets.ISO_8859_1);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }
        
    
    public Map<String, Long>  getTheWords() {
        
        String dir = "f:./data";
    
        return getFiles(dir)
              .flatMap(this::getLines) 
                .flatMap(line -> Arrays.stream(
                        line.toLowerCase().split("[\\(,\\).\\s+]+")))
                .collect(Collectors.groupingBy(word -> word,
                        Collectors.counting())) 
                .entrySet().stream() 
                .sorted(Entry.<String,Long>comparingByValue().reversed().
                        thenComparing(Entry.<String,Long>comparingByKey()))
                .limit(20) // limts the number of entries
                .collect(Collectors.toMap(Entry::getKey, Entry::getValue,
                        (r,u)->r,
                        LinkedHashMap::new));

    }
}

Note. I sorted first on count in reverse order and then, if there was a tie, I sorted alphabetically in normal order.

Answer 2

First, don't mix old File API with new NIO.2 API.

You combine the results from all the files by starting with a Stream of files.

Path dir = Paths.get("data/");
LinkedHashMap<String, Long> top20Words = Files.list(dir)
    .filter(path -> ! Files.isDirectory(path))
    .flatMap(file -> {
        try {
            return Files.lines(file, StandardCharsets.ISO_8859_1);
        } catch (IOException e) {
            e.printStackTrace();
            return Stream.empty();
        }
    })
    // the rest is copied from question, to show context
    .flatMap(line -> Arrays.stream(line.toLowerCase().split("[\\(,\\).\\s+]+")))
    .collect(Collectors.groupingBy(Function.identity(), Collectors.counting())).entrySet().stream()
    .sorted(Entry.comparingByValue(Comparator.reverseOrder()))
    .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
    .collect(Collectors.toMap(Entry::getKey, Entry::getValue, (u, v) -> u, LinkedHashMap::new));
System.out.println(top20Words);