How to read the rest of the lines? - python

Question

I have a file and it has some header lines, e.g.

header1 lines: somehting something
more headers then
somehting something
----

this is where the data starts
yes data... lots of foo barring bar fooing data.
...
...

I've skipped the header lines by looping and running file.readlines(), other than looping and concating the rest of the lines, how else can i read the rest of the lines?

x = """header1 lines: somehting something
more headers then
somehting something
----

this is where the data starts
yes data... lots of foo barring bar fooing data.
...
..."""

with open('test.txt','w') as fout:
  print>>fout, x

fin = open('test.txt','r')
for _ in range(5): fin.readline();
rest = "\n".join([i for i in fin.readline()])

readlines() reads the entire file. Calling it in loops makes no sense. — John Zwinck, Dec 03 '13 at 12:43

Martijn Pieters · Accepted Answer · 2013-12-03T12:48:57.633

4

.readlines() reads the all data in the file, in one go. There are no more lines to read after the first call.

You probably wanted to use .readline() (no s, singular) instead:

with open('test.txt','r') as fin:
    for _ in range(5): fin.readline()
    rest = "\n".join(fin.readlines())

Note that because .readlines() returns a list already, you don't need to loop over the items. You could also just use .read() to read in the remainder of the file:

with open('test.txt','r') as fin:
    for _ in range(5): fin.readline()
    rest = fin.read()

Alternatively, treat the file object as an iterable, and using itertools.islice() slice the iterable to skip the first five lines:

from itertools import islice

with open('test.txt','r') as fin:
    all_but_the_first_five = list(islice(fin, 5, None))

This does produce lines, not one large string, but if you are processing the input file line by line, that usually is preferable anyway. You can loop directly over the slice and handle lines:

with open('test.txt','r') as fin:
    for line in list(islice(fin, 5, None)):
        # process line, first 5 will have been skipped

Don't mix using a file object as an iterable and .readline(); the iteration protocol as implemented by file objects uses an internal buffer to ensure efficiency that .readline() doesn't know about; using .readline() after iteration is liable to return data further on in the file than you expect.

edited Dec 03 '13 at 12:48

answered Dec 03 '13 at 12:43

Martijn Pieters

1,048,767
296
4,058
3,343

ah there is a catch, `readlines()` only read up till an empty line. – alvas Dec 03 '13 at 13:13
@alvas: If you have a file object that behaves differently, you are either adding data at the end or you've found a bug in the buffer handling of the file object in question. – Martijn Pieters Dec 03 '13 at 13:23
it's a bug i think, i check the `len(open(infile,'r').readlines())` and it's far far bigger than `len(readline())x5` + `readlines()` – alvas Dec 03 '13 at 13:37
1

@alvas: Yes, `codecs` has that bug; let me dig that up for you. – Martijn Pieters Dec 03 '13 at 16:00
@alvas: See http://stackoverflow.com/questions/16130904/open-and-codecs-open-in-python-2-7-behave-strangely-different/16155098#16155098 – Martijn Pieters Dec 03 '13 at 16:00
@alvas: I'd use `io.open()` instead of using `codecs` to read unicode from a file. – Martijn Pieters Dec 03 '13 at 16:01

score 1 · Answer 2 · answered Dec 03 '13 at 12:45

1

Skip the first 5 lines:

from itertools import islice

with open('yourfile') as fin:
    data = list(islice(fin, 5, None))
    # or loop line by line still
    for line in islice(fin, 5, None):
        print line

answered Dec 03 '13 at 12:45

Jon Clements

138,671
33
247
280

How to read the rest of the lines? - python

2 Answers2