How to sort records from text file and store into binary?

Question

I have an input text file that is basically a tsv of people. I need to sort all the records (by last name, first name) and then store all the records into a binary file. What I've done so far, is create a DataRecord object which has all the appropriate fields and getter/setters and compareTo. In the main, I've got an ArrayList of type DataRecord for sorting purposes.

public class DataRecord implements Comparable<DataRecord>{

private String lastName, firstName, middleName, suffix, cityOfBirth;
private int monthOfBirth, dayOfBirth, yearOfBirth;
private char gender;


//getters
public String getLastName() { return this.lastName;}
public String getFirstName() { return this.firstName;}
public String getMiddleName() { return this.middleName;}
public String getSuffix() { return this.suffix;}
public String getCityOfBirth() { return this.cityOfBirth;}
public int getMonthOfBirth() { return this.monthOfBirth;}
public int getDayOfBirth() { return this.dayOfBirth;}
public int getYearOfBirth() { return this.yearOfBirth;}
public char getGender() { return this.gender;}

//setters
public void setLastName(String lastName) { this.lastName = lastName;}
public void setFirstName(String firstName) { this.firstName = firstName;}
public void setMiddleName(String middleName) { this.middleName = middleName;}
public void setSuffix(String suffix) { this.suffix = suffix;}
public void setCityOfBirth(String cityOfBirth) { this.cityOfBirth = cityOfBirth;}
public void setMonthOfBirth(int monthOfBirth) { this.monthOfBirth = monthOfBirth;}
public void setDayOfBirth(int dayOfBirth) { this.dayOfBirth = dayOfBirth;}
public void setYearOfBirth(int yearOfBirth) { this.yearOfBirth = yearOfBirth;}
public void setGender(char gender) { this.gender = gender;}

public DataRecord(){

}

//constructor to make copy of record passed in
public DataRecord(DataRecord copyFrom){
    this.lastName = copyFrom.getLastName();
    this.firstName = copyFrom.getFirstName();
    this.middleName = copyFrom.getMiddleName();
    this.suffix = copyFrom.getSuffix();
    this.monthOfBirth = copyFrom.getMonthOfBirth();
    this.dayOfBirth = copyFrom.getDayOfBirth();
    this.yearOfBirth = copyFrom.getYearOfBirth();
    this.gender = copyFrom.getGender();
    this.cityOfBirth = copyFrom.getCityOfBirth();
}

@Override
public int compareTo(DataRecord arg0) {
    // TODO Auto-generated method stub
    int lastNameCompare;

    //check if the last names are the same, if so return the first name comparison
    if ((lastNameCompare = this.getLastName().compareTo(arg0.getLastName())) == 0){
        return this.getFirstName().compareTo(arg0.getFirstName());
    }

    //otherwise return the last name comparison
    return lastNameCompare;

}

public String toString(){
    return this.getLastName() + ' ' + this.getFirstName();
}

}

public class IOController {

  public static void main(String[] args) throws IOException {
    File inputFile; // input file
    RandomAccessFile dataStream = null; // output stream
    ArrayList<DataRecord> records = new ArrayList<DataRecord>();

    BufferedReader reader = new BufferedReader(new FileReader(args[0]));
    try {
        String sb;
        String line = reader.readLine();
        String[] fields;

        // loop through and read all the lines in the input file
        while (line != null) {
            DataRecord currentRecord = new DataRecord();

            // store the current line into a local string
            sb = line;

            // create an array of all the fields
            fields = sb.split("\t");
            // set the fields for the DataRecord object
            currentRecord.setLastName(fields[0]);
            currentRecord.setFirstName(fields[1]);

            // check other fields exist
            if (fields.length >= 3) {
                currentRecord.setMiddleName(fields[2]);
                currentRecord.setSuffix(fields[3]);
                currentRecord.setMonthOfBirth(Integer.parseInt(fields[4]));
                currentRecord.setDayOfBirth(Integer.parseInt(fields[5]));
                currentRecord.setYearOfBirth(Integer.parseInt(fields[6]));
                currentRecord.setGender(fields[7].charAt(0));
                currentRecord.setCityOfBirth(fields[8]);
            }

            // add the current record to the array list of records
            records.add(currentRecord);
            line = reader.readLine();
        }
    } finally {
        reader.close();
      //Collections.sort(records);
    }

    for (int i = 0; i < 5; i++) {
        System.out.println(records.get(i));
    }

}

}

My issue is that if I use a temporary DataRecord (named currentRecord) to read the fields, then add to the ArrayList, I have all the same data in every record in the ArrayList. If I copy that data to another DataRecord object (using a constructor where I pass in a DataRecord), I run out of heapspace.

records.add(new DataRecord(currentRecord));
line = reader.readLine();

Is my mistake using an ArrayList?

Answer 1

You're using the same object reference to add in the ArrayList and are updating it on every iteration. Just create a new instance of the object on each iteration:

while (line != null) {
    DataRecord currentRecord = new DataRecord();
    // rest of the code...
    records.add(currentRecord);
}
//sort the list

As best practice, declare your variables in the narrowest possible scope.

Since you run out of heap space, you may try adding more ram to your process by using -Xmx argument. If you lack of ram in the PC you're executing the process, then use another alternative like splitting the file into small chunks, sort each new file, then using a derivate of merge sort between the data in these files.

Answer 2

Is my mistake using an ArrayList?

No.

Your mistake is one or both of the following:

• Attempting to hold the information contents of a large file in memory at the same time. The alternative is to stream the data; e.g. read record, write record, read, record, write record, etc. (Of course the feasibility of doing that depends on the nature of your "binary" file representation.)

• Attempting to run with a heap that is too small. The java command documentation explains how to increase the heap size, but obviously there are practical limits to that approach.

And for the record, this is also a mistake:

    records.add(currentRecord);

If you do that, you will end up with an list containing (just) N copies of the last record in your CSV input file. If you are going to build an in-memory copy in a list, then you need to create a new DataRecord object for each row.

---

For the record, changing to LinkedList won't help in the long term. The maximum space usage for an ArrayList created by appending to a list created using new ArrayList() is roughly 3 x the size of a reference. For a LinkedList the space usage is 3 x the size of a reference + 2 additional words per entry.