0

I am upgrading from python 2 to python 3. One of the files I am using read_csvwith was causing me to get a MemoryError, was able to get around it but now a new error.

I've tried some suggestions from here How to read a 6 GB csv file with pandas but still getting an error: TypeError: 'TextFileReader' object is not subscriptable

I have narrowed down my columns and added chunksize. My .txt file is now 50k kb's.

Is there anyway around this error? Thanks.

e1 = pd.read_csv(working_dir+"E1.txt",sep=',', chunksize = 10000)
e1['MTM'] = e1['stack_over_flow']

traceback:

Traceback (most recent call last):

  File "<ipython-input-97-99e71d524b4b>", line 1, in <module>
    runfile('C:/AppData/FinRecon/py_code/python3/DataJoin.py', wdir='C:/AppData/FinRecon/py_code/python3')

  File "C:\Users\stack\AppData\Local\Continuum\anaconda3\anaconda3_32bit\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "C:\Users\stack\AppData\Local\Continuum\anaconda3\anaconda3_32bit\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/AppData/FinRecon/py_code/python3/DataJoin.py", line 474, in <module>
    M2()

  File "C:/AppData/FinRecon/py_code/python3/DataJoin.py", line 42, in M2
    e1['MTM'] = e1['stack_over_flow']

    TypeError: 'TextFileReader' object is not subscriptable
excelguy
  • 1,574
  • 6
  • 33
  • 67

1 Answers1

0

One way around this problem is to set nrows parameter in pd.read_csv() function and that way you select subset of data you want to load into the dataframe. Of course, drawback is that you wont be able to see and work with full dataset. Code example:

e1 = pd.read_csv(working_dir+"E1.txt", sep=',', nrows=100000)

Also, if you want to read the file by chunks and process it you may take a look on pandas docs, so you can achieve this by doing:

e1 = pd.read_csv(working_dir+"E1.txt", sep=',', chunks=10000)

for chunk in e1:
    print(chunk)
Lucas Sierota
  • 165
  • 1
  • 2
  • 11