Reading .csv in Python without looping through the whole file?

Question

The only way I've seen Python's csv.reader used is in a for loop, which goes through the whole file without saving past values of the read in variables. I only need to work with 2 consecutive lines of the (enormous) file at a time. Using the csv.reader for loop, I only have 1 line at a time.

Is there a way to use Python's csv module for taking in only one line of a csv file without having to finish reading the file to the end?

I need to set variables to the values in the first line, set a second set of variables to the values of the next line, use the two sets of variables simultaneously for computations, then overwrite the first set of variables with the second set, and read a new line to overwrite the second set.

I don't understand the question. Are you asking whether I am able to store the first line into variables? — mary, Jun 23 '10 at 16:39
yes, what prevents you from storing the first line until the second one arrives, then using both? — SilentGhost, Jun 23 '10 at 16:55
I need to animate an object traveling between the locations provided in each line of data. I wanted to have one while loop that takes lines as it needs them to update velocity, instead of the file-reading for loop which would run a smaller while loop thousands of times. — mary, Jun 23 '10 at 18:22
Near-duplicate of [How to read one single line of csv data in Python?](http://stackoverflow.com/questions/17262256/how-to-read-one-single-line-of-csv-data-in-python) — smci, Aug 27 '16 at 00:25
It looks like that question (June 23 '13) was asked exactly three years after this one (June 23 '10), making that other question the near-duplicate. — mary, Jun 27 '18 at 19:10

score 43 · Accepted Answer · edited Aug 27 '16 at 21:20

43

There's nothing forcing you to use the reader in a loop. Just read the first line, then read the second line.

import csv
rdr = csv.reader(open("data.csv"))
line1 = rdr.next() # in Python 2, or next(rdr) in Python 3
line2 = rdr.next()

edited Aug 27 '16 at 21:20

smci

32,567
20
113
146

answered Jun 23 '10 at 16:10

Bryan Oakley

370,779
53
539
685

Very helpful, thank you. The way I was dividing the line into its columns was: r = csv.reader(open("data.csv")) for col1, col2, col3 in r: x = float( col1 ) y = etc... How do I need to adapt that method using r.next() instead of a for loop? In other words, how do I extract the column entries from the entire line? – mary Jun 23 '10 at 16:49
2

@mary: in my example, line1 is just a list so line1[0] would be the first column, line1[1] the second column, etc. If you want, you could do something like `(col1, col2, col3)=line1` – Bryan Oakley Jun 23 '10 at 16:59
1

In Python 3, use `next(r)` builtin function instead of `r.next()` method. – smci Aug 27 '16 at 00:24
@smci: you might want to explain why that's better. – Bryan Oakley Aug 27 '16 at 01:46
@BryanOakley: because `.next()` method doesn't exist on iterators in Python3 :) – smci Aug 27 '16 at 21:20

score 5 · Answer 2 · answered Jan 15 '15 at 13:13

5

Read CSV:

readCSV = csv.reader(csvFile, delimiter=',')

Read the next row in Python 2.7:

    row = readCSV.next()

Read the next row in Python 3.4:

    row = readCSV.__next__()

answered Jan 15 '15 at 13:13

hatef

5,491
30
43
46

8

I'd recommend using `row = next(readCSV)`. This works in both python 2.7 and 3, and does nothing other than calling the methods you describe. – PeterE Jan 15 '15 at 13:44

score 5 · Answer 3 · answered Jun 23 '10 at 17:04

If you're always looking at exactly two consecutive lines, it sounds to me like you might benefit from using the pairwise recipe. From the itertools module:

from itertools import tee, izip
def pairwise(iterable):
   "s -> (s0,s1), (s1,s2), (s2, s3), ..."
   a, b = tee(iterable)
   next(b, None)
   return izip(a, b)

You would use this like so:

for first_dict, second_dict in pairwise(csv.DictReader(stream)):
    # do stuff with first_dict and second_dict

score 4 · Answer 4 · answered Jun 23 '10 at 16:09

4

The obvious answer seems to be to just store the previous line on each iteration.

>>> for x in csv.DictReader(stream):
...   print prevLine
...   print x
...   prevLine = x
....

answered Jun 23 '10 at 16:09

TK.

417
3
5

3

@TK: this code, as-is, will fail since you don't initialize prevLine. – Bryan Oakley Jun 23 '10 at 16:11

score 4 · Answer 5 · answered Jun 23 '10 at 16:26

4

Blatant stealing from TK... ...mostly the question that remains is, what does the OP want to do with the first and last lines of the file?

prevLine = None

for x in csv.DictReader(stream):
   if prevLine is not None:
       DoWork(prevLine, x)
   else:
       Initialize(x)
   prevLine = x

Finalize(prevLine)

answered Jun 23 '10 at 16:26

dash-tom-bang

17,383
5
46
62

Each line contains positions, then I need to calculate velocities between them and animate an object traveling from one to the next. Once the last position is hit by the object, the program finishes. – mary Jun 23 '10 at 16:36
1

The answer to that question should inform what you need to do with the data then. Either you treat the first and last lines specially (as I have done), or you need to somehow deal with "empty" prev or next lines at the ends. – dash-tom-bang Jun 23 '10 at 16:43

Reading .csv in Python without looping through the whole file?

5 Answers5