1

This is an update

This Out of memory error does not happen when running the code using Jupyter or Spyder UI. Any ideas?

I am loading CSV files about 250 MB in size, when the program runs for the first time it does not throw "out of memory" error but when loading the next csv file it does throw it then.

When I get this error I stopped program and re run it, interestingly I dont get out of memory error but when loading the next csv file, I get the error once again.

My PC has 8 GB left in RAM so I dont see why where is the memory cap?

The code I am using to load the csv file is:

df = pd.read_csv(csvfile, index_col=False, sep=',', low_memory=False)

Traceback Error

Traceback (most recent call last):
  File "C:/Users/jvivas/Dropbox/Private/Personal/Github/CSVFormattingTools/XOMCSVFormattingTool_version2.py", line 34, in <module>
    df = pd.read_csv(csvfile, index_col=False, sep=',', low_memory=False)
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\pandas\io\parsers.py", line 562, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\pandas\io\parsers.py", line 325, in _read
    return parser.read()
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\pandas\io\parsers.py", line 815, in read
    ret = self._engine.read(nrows)
  File "C:\Program Files (x86)\Python35-32\lib\site-packages\pandas\io\parsers.py", line 1314, in read
    data = self._reader.read(nrows)
  File "pandas\parser.pyx", line 808, in pandas.parser.TextReader.read (pandas\parser.c:8643)
  File "pandas\parser.pyx", line 896, in pandas.parser.TextReader._read_rows (pandas\parser.c:9772)
  File "pandas\parser.pyx", line 1865, in pandas.parser.raise_parser_error (pandas\parser.c:23295)
pandas.io.common.CParserError: Error tokenizing data. C error: out of memory

Any help is appreciated

Thanks

racekiller
  • 115
  • 1
  • 2
  • 13
  • is [this](http://stackoverflow.com/questions/30245640/pandas-read-csv-out-of-memory) any help? – Paul Rooney Sep 26 '16 at 23:01
  • 2
    I thought pandas was able to load big file without this kind of issue – racekiller Sep 27 '16 at 01:07
  • Does it fail just loading the second file after restart? also I'm assuming you're using 64-bit OS, python etc. – EdChum Sep 27 '16 at 08:01
  • Correct @EdChum the OS is 64bit and python is 32 bit. Yes few seconds later of starting the loading it throws the error, from 1 to 2 seconds. thanks – racekiller Sep 27 '16 at 14:18

0 Answers0