Retrieve 10 random lines from a file

Question

I have a text file which is 10k lines long and I need to build a function to extract 10 random lines each time from this file. I already found how to generate random numbers in Python with numpy and also how to open a file but I don't know how to mix it all together. Please help.

Answer 1

If you know that your file is exactly 10k lines long then you can use linecache:

import random
import linecache

def random_lines(filename)
    idxs = random.sample(range(10000), 10)
    return [linecache.getline(filename, i) for i in idxs]

This returns a list with 10 random lines which you can print with:

for line in random_lines('file.txt'):
    print(line)

Answer 2

If you know how many lines long the file is, you can use linecache, as the other answers suggest. But just knowing that it's 10K in size tells you nothing about how many lines long it is.

If you know the file is small enough to fit into memory—and a 10K file is easily small enough—just read it into memory:

import random

with open('file.txt') as f:
    lines = list(f)
for i in range(10):
    print(random.choice(lines))

But what if you don't know how long the file is, and can't afford to read it all into memory? Then you'll need to do two passes:

import linecache
import random

with open('file.txt') as f:
    linecount = sum(1 for line in f)
for i in range(10):
    print(linecache.getline('file.txt', random.range(linecount)))

---

Note that both are going to leave newlines at the end of each line. If you want to get rid of these, you can change the first example from list(f) to [line.rstrip() for line in f], or just call rstrip() in the print, or use end='' (Python 3.x) or a trailing comma (Python 2.x) in the print. For the linecache example, the first obviously doesn't work, but you can still do either of the others.

---

Also note that I used the stdlib random library instead of using numpy here. If you're just generating 10 random numbers to be used in normal Python code, there's no reason to use numpy. (On the other hand, if you do have a good reason to use numpy here, you may want to read the lines into a pandas table and apply the random indices to that.)

Answer 3

You may use this code which doesn't care about the file length, however in rare occasions you may get duplicates :

from random import choice
lines = [a.strip() for a in open("yourfile").readlines()]
result = [choice(lines) for a in range(10)]

result is a list containing 10 lines chosen randomly from the file named yourfile.

Answer 4

It is possible to do the job with one pass and without loading the entire file into memory as well. Though the code itself is going to be much more complicated and mostly unneeded unless the file is HUGE.

The trick is the following:

Suppose we only need one random line, then first save first line into a variable, then for ith line, replace the currently with probability 1/i. Return the saved line when reaching end of file. For 10 random lines, then have an list of 10 element and do the process 10 times for each line in the file.

Answer 5

try linecache:

import linecache
#put your 3 randoms into an array in whichever way you are doing it
lines = [3,45,678]  #use your existing code here
for i in lines:
    linecache.getline('/etc/file', i)

Answer 6

If you don't know the number of lines of your file, you can count them with, for example, this code:

line_count = 0
with open(filename) as file:
    for line in file:
        line_count += 1

Then, you will be able to generate random numbers within the range [0, line_count) :

import random
lines_to_read = []
for i in range(10):
    line = random.randint(0, line_count - 1)
    lines_to_read.append(line)

And finally, read the file again, select those lines randomly chosen, and do whatever you want with them, for example, print them:

with open(filename) as file:
    for index, line in enumerate(file):
        if index in lines_to_read:
            print line

I hope it helps you! Cheers!