How to sort a large text file alphabetically?

Question

So I have a text file and I need to sort the lines alphabetically. Example input:

This is the first sentence
A sentence here as well
But how do I reorder them?

Output:

A sentence here as well
But how do I reorder them?
This is the first sentence

Here's the thing: This file is so large, I don't have enough RAM to actually split it into a list/array. I tried to use Python's built-in sorted() function and the process got killed.

To give you an idea:

wc -l data
21788172 data

You can use shell `sort` program. You probably don't need to do everything with python. — Suor, Apr 07 '14 at 02:02
depending on your hardware [MapReduce](http://en.wikipedia.org/wiki/MapReduce) could be interesting for you. — vikingosegundo, Apr 07 '14 at 02:16

score 5 · Answer 1 · edited May 23 '17 at 12:14

5

It sounds like you need to do a merge-sort: divide the file into blocks, sort each block, then merge the sorted blocks back together. See Python class to merge sorted files, how can this be improved?

edited May 23 '17 at 12:14

Community

1
1

answered Apr 07 '14 at 01:59

Hugh Bothwell

55,315
8
84
99

score 1 · Answer 2 · answered Apr 07 '14 at 02:34

Similarly to what Hugh recommended (but different in that this isn't a pure-Python solution), you could sort the file in chunks. E.g., split the file into 26 other files--A.txt, B.txt, C.txt, etc. Sort each of those individually and then combine them to get the final result.

Main thing to keep in mind is that the first pass through the source file is merely to divvy up the lines to their constituent first letters. Only after that do you run the sorts through each file. A simple cat A.txt B.txt ... will handle the rest.

How to sort a large text file alphabetically?

2 Answers2