load a certain number of rows from csv with numpy

Question

I have a very long file and I only need parts, a slice, of it. There is new data coming in so the file will potentially get longer.

To load the data from the CSV I use numpy.genfromtxt

    np.genfromtxt(filename, usecols={col}, delimiter=",", skip_header=skip_head)

This cuts off a certain parts of the file in the beginning which already substantially speeds up the process of loading the data. But I can't use skip_footer in the end to cut off the part after my slice that I want to use.

What I want is to only load a certain number of rows. e.g. lets say I skip the first 100 rows, then load the next 50 rows and skip the rest afterwards.

edit: I am using Python 3.4
edit: sample file: http://www.file-upload.net/download-10819938/sample.txt.html

Padraic Cunningham · Accepted Answer · 2015-08-05T16:24:11.533

4

You could get the slice using itertools, taking the column using itemgetter:

import  numpy as np
from operator import itemgetter
import csv
with open(filename) as f:
   from itertools import islice,imap
   r = csv.reader(f)
   np.genfromtxt(imap(itemgetter(1),islice(r,  start, end+1)))

For python3, you can use fromiter with the code above you need to specify the dtype:

import numpy as np
from operator import itemgetter
import csv
with open("sample.txt") as f:
   from itertools import islice
   r = csv.reader(f)
   print(np.fromiter(map(itemgetter(0), islice(r,  start, end+1)), dtype=float))

As in the other answer you can also pass the islice object directly to genfromtxt but for python3 you will need to open the file in binary mode:

with open("sample.txt", "rb") as f:
    from itertools import islice
    print(np.genfromtxt(islice(f, start, end+1), delimiter=",", usecols=cols))

Interestingly, for multiple columns using itertools.chain and reshaping is over twice as efficient if all your dtypes are the same:

from itertools import islice,chain
with open("sample.txt") as f:
   r = csv.reader(f)
   arr =np.fromiter(chain.from_iterable(map(itemgetter(0, 4, 10), 
                                            islice(r,  4, 10))), dtype=float).reshape(6, -1)

On you sample file:

In [27]: %%timeit
with open("sample.txt", "rb") as f:
    (np.genfromtxt(islice(f, 4, 10), delimiter=",", usecols=(0, 4, 10),dtype=float))
   ....: 

10000 loops, best of 3: 179 µs per loop

In [28]: %%timeit
with open("sample.txt") as f:
   r = csv.reader(f)                                                               (np.fromiter(chain.from_iterable(map(itemgetter(0, 4, 10), islice(r,  4, 10))), dtype=float).reshape(6, -1))

10000 loops, best of 3: 86 µs per loop

edited Aug 05 '15 at 16:24

answered Aug 05 '15 at 13:13

Padraic Cunningham

176,452
29
245
321

hm i can't import imap. I think it's because i'm using python 3.4 – BlueScr33n Aug 05 '15 at 13:54
1

Just use map with python 3 – Padraic Cunningham Aug 05 '15 at 13:56
do you mean to just replace imap with map? – BlueScr33n Aug 05 '15 at 14:24
had a mistake in the original code anyway, need to use the csv module to split the data. Can you add a sample of your file so I can run it locally? – Padraic Cunningham Aug 05 '15 at 14:37
now i get the TypeError: 'Can't convert 'bytes' object to str implicitly' – BlueScr33n Aug 05 '15 at 15:59
@BlueScr33n, for fromiter you won't,you can pass the islice object directly but you need to open the file in binary mode – Padraic Cunningham Aug 05 '15 at 16:00
ok I hope this is the last problem I am encountering. It tells me that the required argument 'dtype' (pos 2) is not found. Since when is dtype required? Also I still get this error even when I specify dtype after np.fromiter – BlueScr33n Aug 05 '15 at 16:21
You should not need to specify the dtype with genfromtxt, it is optional. If you don't provide it numpy will guess the datatype. What version of numpy as you using? – Padraic Cunningham Aug 05 '15 at 16:25
all right opening it in binary mode seems to be working – BlueScr33n Aug 05 '15 at 17:22
alright seems like I missunderstood something. I thought the end would be the number of columns that are loaded after the skipped rows. Now I get it :) I installed python with Anaconda and my numpyversion is 1.9.2 – BlueScr33n Aug 05 '15 at 17:35
Nope, end is where you want to stop, you just need +1 as it is half open. It is all working? – Padraic Cunningham Aug 05 '15 at 17:55

score 2 · Answer 2 · edited May 23 '17 at 10:26

2

Following this example, you should be able to use itertools.islice, without needing imap, map or csv.reader:

import numpy as np
import itertools

with open('sample.txt') as f:
    # this will skip 100 lines, then read the next 50
    d=np.genfromtxt(itertools.islice(f,100,150),delimiter=',',usecols={cols})

edited May 23 '17 at 10:26

Community

1
1

answered Aug 05 '15 at 15:33

tmdavison

64,360
12
187
165

I tried, I get the TypeError: 'Can't convert 'bytes' object to str implicitly' – BlueScr33n Aug 05 '15 at 15:58

score 2 · Answer 3 · answered Mar 03 '19 at 12:26

Starting Numpy 1.10, np.genfromtxt takes an optional parameter max_rows which limits the number of lines to read.

Combined with the other optional parameter skip_header, you can select a slice of your file (for instance lines 100 to 150):

import numpy as np

np.loadtxt('file.txt', skip_header=100, max_rows=50)

load a certain number of rows from csv with numpy

3 Answers3

Linked

Related