7

I am processing a csv-file which is 2.5 GB big. The 2.5 GB table looks like this:

columns=[ka,kb_1,kb_2,timeofEvent,timeInterval]
0:'3M' '2345' '2345' '2014-10-5',3000
1:'3M' '2958' '2152' '2015-3-22',5000
2:'GE' '2183' '2183' '2012-12-31',515
3:'3M' '2958' '2958' '2015-3-10',395
4:'GE' '2183' '2285' '2015-4-19',1925
5:'GE' '2598' '2598' '2015-3-17',1915

And I want to groupby ka and kb_1 to get the result like this:

columns=[ka,kb,errorNum,errorRate,totalNum of records]
'3M','2345',0,0%,1
'3M','2958',1,50%,2
'GE','2183',1,50%,2
'GE','2598',0,0%,1

(definition of error Record: when kb_1 != kb_2, the corresponding record is treated as abnormal record )

My computer, which is ubuntu 12.04, has 16 GB memory and free -m returns

             total       used       free     shared    buffers     cached
Mem:        112809      14476      98333          0        128      10823
-/+ buffers/cache:       3524     109285
Swap:    

    0          0          0

My python file is called bigData.py

import pandas as pd
import numpy as np

import sys,traceback,os
cksize=98333 # or 1024, either chunk size didn't work at all
try:
    dfs = pd.DataFrame()
    reader=pd.read_table('data/petaJoined.csv', chunksize=cksize)  

    for chunk in reader:#when executed this line,error occur!
        pass
        #temp=tb_createTopRankTable(chunk)
        #dfs.append(temp)
        #df=tb_createTopRankTable(dfs)
   except:
    traceback.print_exc(file=sys.stdout)

ipdb> pd.__version__
'0.16.0'

I use the following command to monitor the memory usage:

top 
ps -C python -o %cpu,%mem,cmd

Since it takes about 2 seconds to crash, so I can see the mem usage had reached 90% some time, and CPU usage reached 100%

When I excecute python bigData.py, the following error generate:

/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
  from pkg_resources import resource_stream
/usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
  from pkg_resources import resource_stream
Traceback (most recent call last):
  File "bigData.py", line 10, in <module>
    for chunk in reader:
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
    yield self.read(self.chunksize)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
    ret = self._engine.read(nrows)
  File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
    data = self._reader.read(nrows)
  File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
  File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
  File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
  File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
  File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
CParserError: Error tokenizing data. C error: out of memory
Segmentation fault (core dumped)

or

     /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module dateutil was already imported from /usr/local/lib/python2.7/dist-packages/dateutil/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
      from pkg_resources import resource_stream
    /usr/local/lib/python2.7/dist-packages/pytz/__init__.py:29: UserWarning: Module pytz was already imported from /usr/local/lib/python2.7/dist-packages/pytz/__init__.pyc, but /usr/lib/python2.7/dist-packages is being added to sys.path
      from pkg_resources import resource_stream
    Traceback (most recent call last):
      File "bigData.py", line 10, in <module>
        for chunk in reader:
      File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 691, in __iter__
        yield self.read(self.chunksize)
      File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 715, in read
        ret = self._engine.read(nrows)
      File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py", line 1164, in read
        data = self._reader.read(nrows)
      File "pandas/parser.pyx", line 758, in pandas.parser.TextReader.read (pandas/parser.c:7411)
      File "pandas/parser.pyx", line 792, in pandas.parser.TextReader._read_low_memory (pandas/parser.c:7819)
      File "pandas/parser.pyx", line 833, in pandas.parser.TextReader._read_rows (pandas/parser.c:8268)
      File "pandas/parser.pyx", line 820, in pandas.parser.TextReader._tokenize_rows (pandas/parser.c:8142)
      File "pandas/parser.pyx", line 1758, in pandas.parser.raise_parser_error (pandas/parser.c:20728)
    CParserError: Error tokenizing data. C error: out of memory
    *** glibc detected *** python: free(): invalid pointer: 0x00007f750d2a4c0e ***
    ====== Backtrace: ========
    /lib/x86_64-linux-gnu/libc.so.6(+0x7db26)[0x7f7511529b26]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0x4d5a1)[0x7f750d29d5a1]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_cleanup+0x15)[0x7f750d29de45]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(parser_free+0x9)[0x7f750d29e039]
    /usr/local/lib/python2.7/dist-packages/pandas/parser.so(+0xb43e)[0x7f750d25b43e]
   ....
    python(PyDict_SetItem+0x49)[0x577749]
    python(_PyModule_Clear+0x149)[0x4cafb9]
    python(PyImport_Cleanup+0x477)[0x4cb4f7]
    python(Py_Finalize+0x18e)[0x549f0e]
    python(Py_Main+0x3bc)[0x56b56c]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f75114cd76d]
    python[0x41bb11]
    ======= Memory map: ========
    00400000-00670000 r-xp 00000000 08:01 26612                              /usr/bin/python2.7
    0086f000-00870000 r--p 0026f000 08:01 26612                              /usr/b.......
    008d9000-008eb000 rw-p 00000000 00:00 0 
    01ddb000-036f7000 rw-p 00000000 00:00 0                                  [heap]
    7f748c179000-7f74cc17a000 rw-p 00000000 00:00 0 
    7f7504000000-7f7504021000 rw-p 00000000 00:00 0 
    7f7504021000-7f7508000000 ---p 00000000 00:00 0 
    7f750bf83000-7f750c285000 rw-p 00000000 00:00 0 
    7f750c285000-7f750c586000 rw-p 00000000 00:00 0 
    7f750c586000-7f750c707000 rw-p 00000000 00:00 0 
    7f750c707000-7f750c711000 r-xp 00000000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c711000-7f750c911000 ---p 0000a000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c911000-7f750c912000 r--p 0000a000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c912000-7f750c913000 rw-p 0000b000 08:01 533205                     /usr/local/lib/python2.7/dist-packages/pandas/_testing.so
    7f750c913000-7f750c914000 rw-p 00000000 00:00 0 
    7f750c914000-7f750c918000 r-xp 00000000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750c918000-7f750cb17000 ---p 00004000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750cb17000-7f750cb18000 r--p 00003000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750cb18000-7f750cb19000 rw-p 00004000 08:01 2331                       /lib/x86_64-linux-gnu/libuuid.so.1.3.0
    7f750cb19000-7f750cb34000 r-xp 00000000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cb34000-7f750cd33000 ---p 0001b000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cd33000-7f750cd34000 r--p 0001a000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cd34000-7f750cd38000 rw-p 0001b000 08:01 533071                     /usr/local/lib/python2.7/dist-packages/pandas/msgpack.so
    7f750cd38000-7f750d039000 rw-p 00000000 00:00 0 
    7f750d039000-7f750d04e000 r-xp 00000000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d04e000-7f750d24e000 ---p 00015000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d24e000-7f750d24f000 r--p 00015000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d24f000-7f750d250000 rw-p 00016000 08:01 533070                     /usr/local/lib/python2.7/dist-packages/pandas/json.so
    7f750d250000-7f750d2a9000 r-xp 00000000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d2a9000-7f750d4a8000 ---p 00059000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d4a8000-7f750d4a9000 r--p 00058000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d4a9000-7f750d4af000 rw-p 00059000 08:01 533270                     /usr/local/lib/python2.7/dist-packages/pandas/parser.so
    7f750d4af000-7f750d591000 r-xp 00000000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d591000-7f750d790000 ---p 000e2000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d790000-7f750d798000 r--p 000e1000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d798000-7f750d79a000 rw-p 000e9000 08:01 49584                      /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
    7f750d79a000-7f750d7af000 rw-p 00000000 00:00 0 
    7f750d7af000-7f750d7f1000 r-xp 00000000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d7f1000-7f750d9f1000 ---p 00042000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d9f1000-7f750d9f3000 r--p 00042000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d9f3000-7f750d9f4000 rw-p 00044000 08:01 530477                     /usr/lib/pyshared/python2.7/matplotlib/_path.so
    7f750d9f4000-7f750da2d000 r-xp 00000000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750da2d000-7f750dc2c000 ---p 00039000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750dc2c000-7f750dc2d000 r--p 00038000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750dc2d000-7f750dc31000 rw-p 00039000 08:01 533269                     /usr/local/lib/python2.7/dist-packages/pandas/_sparse.so
    7f750dc31000-7f750dc7d000 r-xp 00000000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750dc7d000-7f750de7c000 ---p 0004c000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750de7c000-7f750de7d000 r--p 0004b000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750de7d000-7f750de86000 rw-p 0004c000 08:01 533447                     /usr/local/lib/python2.7/dist-packages/pandas/_period.so
    7f750de86000-7f750de87000 rw-p 00000000 00:00 0 
    7f750de87000-7f750debc000 r-xp 00000000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750debc000-7f750e0bb000 ---p 00035000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750e0bb000-7f750e0bc000 r--p 00034000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750e0bc000-7f750e0c0000 rw-p 00035000 08:01 533203                     /usr/local/lib/python2.7/dist-packages/pandas/index.so
    7f750e0c0000-7f750e295000 r-xp 00000000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e295000-7f750e494000 ---p 001d5000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e494000-7f750e495000 r--p 001d4000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e495000-7f750e4a9000 rw-p 001d5000 08:01 533278                     /usr/local/lib/python2.7/dist-packages/pandas/algos.so
    7f750e4a9000-7f750e4ac000 rw-p 00000000 00:00 0 
    7f750e4ac000-7f750e4b2000 r-xp 00000000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e4b2000-7f750e6b1000 ---p 00006000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e6b1000-7f750e6b2000 r--p 00005000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e6b2000-7f750e6b4000 rw-p 00006000 08:01 48831                      /usr/lib/python2.7/lib-dynload/_csv.so
    7f750e6b4000-7f750e782000 r-xp 00000000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e782000-7f750e981000 ---p 000ce000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e981000-7f750e982000 r--p 000cd000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e982000-7f750e990000 rw-p 000ce000 08:01 533449                     /usr/local/lib/python2.7/dist-packages/pandas/lib.so
    7f750e990000-7f750e992000 rw-p 00000000 00:00 0 
    7f750e992000-7f750ea8f000 r-xp 00000000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750ea8f000-7f750ec8e000 ---p 000fd000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750ec8e000-7f750ec8f000 r--p 000fc000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750ec8f000-7f750eca1000 rw-p 000fd000 08:01 533271                     /usr/local/lib/python2.7/dist-packages/pandas/tslib.so
    7f750eca1000-7f750eca4000 rw-p 00000000 00:00 0 
    7f750eca4000-7f750ecc5000 r-xp 00000000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750ecc5000-7f750eec4000 ---p 00021000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750eec4000-7f750eec5000 r--p 00020000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750eec5000-7f750eec9000 rw-p 00021000 08:01 48837                      /usr/lib/python2.7/lib-dynload/_ctypes.so
    7f750eec9000-7f750eeca000 rw-p 00000000 00:00 0 
    7f750eeca000-7f750ef24000 r-xp 00000000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750ef24000-7f750f123000 ---p 0005a000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750f123000-7f750f124000 r--p 00059000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750f124000-7f750f15c000 rw-p 0005a000 08:01 532046                     /usr/local/lib/python2.7/dist-packages/numpy/random/mtrand.so
    7f750f15c000-7f750f15d000 rw-p 00000000 00:00 0 
    7f750f15d000-7f750f166000 r-xp 00000000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f166000-7f750f365000 ---p 00009000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f365000-7f750f366000 r--p 00008000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f366000-7f750f367000 rw-p 00009000 08:01 532085                     /usr/local/lib/python2.7/dist-packages/numpy/fft/fftpack_lite.so
    7f750f367000-7f750f368000 r-xp 00000000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f368000-7f750f567000 ---p 00001000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f567000-7f750f568000 r--p 00000000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f568000-7f750f569000 rw-p 00001000 08:01 48818                      /usr/lib/python2.7/lib-dynload/future_builtins.so
    7f750f569000-7f750f588000 r-xp 00000000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f588000-7f750f787000 ---p 0001f000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f787000-7f750f788000 r--p 0001e000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f788000-7f750f791000 rw-p 0001f000 08:01 48815                      /usr/lib/python2.7/lib-dynload/_io.so
    7f750f791000-7f750f907000 r-xp 00000000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750f907000-7f750fb06000 ---p 00176000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750fb06000-7f750fb07000 r--p 00175000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750fb07000-7f750fb08000 rw-p 00176000 08:01 532132                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/_umath_linalg.so
    7f750fb08000-7f750fba4000 rw-p 00000000 00:00 0 
    7f750fba4000-7f750fd01000 r-xp 00000000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750fd01000-7f750ff00000 ---p 0015d000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750ff00000-7f750ff01000 r--p 0015c000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750ff01000-7f750ff02000 rw-p 0015d000 08:01 532128                     /usr/local/lib/python2.7/dist-packages/numpy/linalg/lapack_lite.so
    7f750ff02000-7f750ff9d000 rw-p 00000000 00:00 0 
    7f750ff9d000-7f750ffa3000 r-xp 00000000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f750ffa3000-7f75101a2000 ---p 00006000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f75101a2000-7f75101a3000 r--p 00005000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f75101a3000-7f75101a4000 rw-p 00006000 08:01 532067                     /usr/local/lib/python2.7/dist-packages/numpy/lib/_compiled_base.so
    7f7510265000-7f751028f000 r-xp 00000000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f751028f000-7f751048e000 ---p 0002a000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f751048e000-7f751048f000 r--p 00029000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f751048f000-7f7510491000 rw-p 0002a000 08:01 532108                     /usr/local/lib/python2.7/dist-packages/numpy/core/scalarmath.so
    7f7510491000-7f75104d2000 rw-p 00000000 00:00 0 
    7f75104d2000-7f75104d5000 r-xp 00000000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75104d5000-7f75106d4000 ---p 00003000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75106d4000-7f75106d5000 r--p 00002000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75106d5000-7f75106d7000 rw-p 00003000 08:01 48833                      /usr/lib/python2.7/lib-dynload/_heapq.so
    7f75106d7000-7f751073e000 r-xp 00000000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f751073e000-7f751093d000 ---p 00067000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f751093d000-7f751093e000 r--p 00066000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f751093e000-7f7510942000 rw-p 00067000 08:01 532118                     /usr/local/lib/python2.7/dist-packages/numpy/core/umath.so
    7f7510942000-7f7510944000 rw-p 00000000 00:00 0 
    7f7510944000-7f7510958000 r-xp 00000000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510958000-7f7510b57000 ---p 00014000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510b57000-7f7510b58000 r--p 00013000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510b58000-7f7510b5c000 rw-p 00014000 08:01 48804                      /usr/lib/python2.7/lib-dynload/datetime.so
    7f7510b5c000-7f7510caf000 r-xp 00000000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510caf000-7f7510eae000 ---p 00153000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510eae000-7f7510eb0000 r--p 00152000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510eb0000-7f7510ebd000 rw-p 00154000 08:01 532106                     /usr/local/lib/python2.7/dist-packages/numpy/core/multiarray.so
    7f7510ebd000-7f7510ecf000 rw-p 00000000 00:00 0 
    7f7510ecf000-7f7510f08000 r-xp 00000000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f7510f08000-7f7511107000 ---p 00039000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f7511107000-7f7511108000 r--p 00038000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f7511108000-7f751110c000 rw-p 00039000 08:01 533450                     /usr/local/lib/python2.7/dist-packages/pandas/hashtable.so
    7f751110c000-7f751110d000 rw-p 00000000 00:00 0 
    7f751110d000-7f7511296000 r--p 00000000 08:01 58562                      /usr/lib/locale/locale-archive
    7f7511296000-7f75112ab000 r-xp 00000000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75112ab000-7f75114aa000 ---p 00015000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75114aa000-7f75114ab000 r--p 00014000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75114ab000-7f75114ac000 rw-p 00015000 08:01 2312                       /lib/x86_64-linux-gnu/libgcc_s.so.1
    7f75114ac000-7f7511660000 r-xp 00000000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f7511660000-7f751185f000 ---p 001b4000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f751185f000-7f7511863000 r--p 001b3000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f7511863000-7f7511865000 rw-p 001b7000 08:01 2327                       /lib/x86_64-linux-gnu/libc-2.15.so
    7f7511865000-7f751186a000 rw-p 00000000 00:00 0 
    7f751186a000-7f7511965000 r-xp 00000000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511965000-7f7511b64000 ---p 000fb000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511b64000-7f7511b65000 r--p 000fa000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511b65000-7f7511b66000 rw-p 000fb000 08:01 2400                       /lib/x86_64-linux-gnu/libm-2.15.so
    7f7511b66000-7f7511b7c000 r-xp 00000000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511b7c000-7f7511d7b000 ---p 00016000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511d7b000-7f7511d7c000 r--p 00015000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511d7c000-7f7511d7d000 rw-p 00016000 08:01 2288                       /lib/x86_64-linux-gnu/libz.so.1.2.3.4
    7f7511d7d000-7f7511f2f000 r-xp 00000000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f7511f2f000-7f751212e000 ---p 001b2000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f751212e000-7f7512149000 r--p 001b1000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f7512149000-7f7512154000 rw-p 001cc000 08:01 2279                       /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
    7f7512154000-7f7512158000 rw-p 00000000 00:00 0 
    7f7512158000-7f75121ac000 r-xp 00000000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75121ac000-7f75123ac000 ---p 00054000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75123ac000-7f75123af000 r--p 00054000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75123af000-7f75123b6000 rw-p 00057000 08:01 2393                       /lib/x86_64-linux-gnu/libssl.so.1.0.0
    7f75123b6000-7f75123b8000 r-xp 00000000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75123b8000-7f75125b7000 ---p 00002000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75125b7000-7f75125b8000 r--p 00001000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75125b8000-7f75125b9000 rw-p 00002000 08:01 2283                       /lib/x86_64-linux-gnu/libutil-2.15.so
    7f75125b9000-7f75125bb000 r-xp 00000000 08:01 2406                                            

/lib/x86_64-linux-gnu/ld-2.15.so
    7f7512a2d000-7f7512b31000 rw-p 00000000 00:00 0 
    7f7512b62000-7f7512bea000 rw-p 00000000 00:00 0 
    7f7512bf7000-7f7512bf9000 rw-p 00000000 00:00 0 
    7f7512bf9000-7f7512bfa000 rwxp 00000000 00:00 0 
    7f7512bfa000-7f7512bfc000 rw-p 00000000 00:00 0 
    7f7512bfc000-7f7512bfd000 r--p 00022000 08:01 2260                       /lib/x86_64-linux-gnu/ld-2.15.so
    7f7512bfd000-7f7512bff000 rw-p 00023000 08:01 2260                       /lib/x86_64-linux-gnu/ld-2.15.so
    7ffcf454c000-7ffcf4585000 rw-p 00000000 00:00 0                          [stack]
    7ffcf459b000-7ffcf459d000 r-xp 00000000 00:00 0                          [vdso]
    ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
    Aborted (core dumped)

with below code, there is no memory problem, but what can the below code do , I mean doing group by and data aggregation

with open("data/petaJoined.csv", "r") as content:
    for line in content:
        #print line
        pass
     #do stuff with line` 
    content.close()

Anyone knows what is happening?

Actually I want to reach the result shown in Pandas read csv out of memory

Maybe there will be a solution?

Note I already use read csv by chunk, but still there is memory error

Then, I changed the chunk size to have my bigData.py file in another way

import pandas as pd
import numpy  as np
import sys, traceback, os
import etl2                                    # my self processing flow
reload(etl2)
def iter_chunks(n,df):
    while True:
        try:
           yield df.get_chunk(n)
        except StopIteration:
            break
cksize=5
try:
    dfs = pd.DataFrame()
    reader=pd.read_table( 'data/petaJoined.csv',
                          chunksize   = cksize,
                          low_memory  = False,
                          iterator    = True
                          )                    # choose as appropriate
    for chunk in iter_chunks(cksize,reader):
        temp=etl2.tb_createTopRankTable(chunk)
        dfs.append(temp)
    df=tb_createTopRankTable(dfs)
    #
    # for chunk in reader:
    #     pass
    # temp=tb_createTopRankTable(chunk)
    # dfs.append(temp)
    # df=tb_createTopRankTable(dfs)
except:
    traceback.print_exc(file=sys.stdout)

Still, there will be segmentation error after running for sometime

def tb_createTopRankTable(df):
    try:
        key='name1'
        key2='name2'
        df2 = df.groupby([key,key2])['isError'].agg({ 'errorNum':  'sum','totalParcel': 'count' })
        df2['errorRate'] = df2['errorNum'] / df2['totalParcel']
        return df2
Community
  • 1
  • 1
sunxd
  • 743
  • 1
  • 9
  • 24
  • the same error if I put the chunk size to be 1024 – sunxd May 15 '15 at 10:18
  • 2
    It will be more memory efficient to read the file line by line, instead of loading it fully using pandas.read_tables before. – Gabriel Hautclocq May 15 '15 at 10:24
  • @Gabriel Hautclocq ,how to do that? – sunxd May 15 '15 at 10:25
  • @PadraicCunningham reader=pd.read_table('data/petaJoined.csv', chunksize=cksize,low_memory=False) didn't help – sunxd May 15 '15 at 10:27
  • `with open("data/petaJoined.csv", "r") as content:` ` for line in content:` ` #do stuff with line` `content.close()` – Gabriel Hautclocq May 15 '15 at 10:28
  • Also what version of pandas are you using? – Padraic Cunningham May 15 '15 at 10:39
  • @PadraicCunningham do you mean this? What information do you need from top? 367 root 20 0 0 0 0 S 0 0.0 0:25.64 kworker/5:1 3556 sunxd 20 0 90376 1820 912 S 0 0.0 0:00.35 sshd 4946 sunxd 20 0 9036 1320 860 R 0 0.0 0:00.08 top 1 root 20 0 24428 2344 1336 S 0 0.0 0:19.10 init – sunxd May 15 '15 at 10:41
  • @PadraicCunningham, only 0.4 of the total memory , I can't paste the info since it elapses too quickly – sunxd May 15 '15 at 10:47
  • @PadraicCunningham, 1."how quickly does the code error?" about two seconds , the cursor blinked five times and then the error occur, exactly at the line "for chunk in reader" 2. top -p proc_pid, how can I know the pid of it since it occurred so quickly ? 3. Panda 0.16.0 – sunxd May 15 '15 at 10:57
  • @sunxd, I presumed it was taking a it was taking a while longer than two seconds before you hit a memory error, I suggested top as you can see the actual memory being used to confirm it is actually a memory issue. – Padraic Cunningham May 15 '15 at 11:00
  • @Rajkumar Kumawat, your code never meet such problem? – sunxd May 15 '15 at 11:27
  • @PadraicCunningham, 90% memory usage as I could capture using top, the way I did is type top and open the monitor and then execute my bigData.py script and see how top changes, it's hard for me to insert the pid of my python task to top since it only goes for 2 seconds – sunxd May 15 '15 at 11:29
  • what if you set the chunk size to a very small number? `chunksize=5` – Padraic Cunningham May 15 '15 at 11:51
  • @PadraicCunningham it just took a bit longer to crash. I guess it's not a problem of the chunksize but instead the big number of chunks! since it crashed at the line :for chunk in reader:, it is actually the for loop ruined the memory – sunxd May 15 '15 at 12:00
  • the for loop with a chunksize of `98333` will read that many rows at once,*chunksize: number of rows to be used to “chunk” a file into pieces.* If you had very big rows that would be a lot of memory. If you use `iterator=True` and `reader.get_chunk(5)` what happens? – Padraic Cunningham May 15 '15 at 12:02
  • @Padraic, this time it worked, but what I want to do to the 2.5G csv is to group by two keys, the exact requirement is shown in http://stackoverflow.com/questions/30245640/pandas-read-csv-out-of-memory. How to know reader.get_chunk(5)has reached the end of the file? – sunxd May 15 '15 at 12:23
  • easy, I will add the code to pastebin, you did also try `chunksize=5`? – Padraic Cunningham May 15 '15 at 12:31
  • http://pastebin.com/jj3bj2X7 – Padraic Cunningham May 15 '15 at 12:37
  • @PadraicCunningham, I did set the chunksize to 5, but after using the code in pastebin.com, it run out of memory after maybe 10 seconds – sunxd May 15 '15 at 13:17
  • @PadraicCunningham, see the latest edit at the bottom part of my question, is that the right way of using your code? – sunxd May 15 '15 at 13:23
  • Yes, the error must be because you are running out of memory creating the Dataframe. what is `etl2.tb_createTopRankTable`? – Padraic Cunningham May 15 '15 at 13:36
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/77909/discussion-between-sunxd-and-padraic-cunningham). – sunxd May 15 '15 at 14:43

2 Answers2

2

Based on your snippet, when reading line-by-line.

I assume that kb_2 is the error indicator,

groups={}
with open("data/petaJoined.csv", "r") as large_file:
    for line in large_file:
        arr=line.split('\t')
        #assuming this structure: ka,kb_1,kb_2,timeofEvent,timeInterval
        k=arr[0]+','+arr[1]
        if not (k in groups.keys())
            groups[k]={'record_count':0, 'error_sum': 0}
        groups[k]['record_count']=groups[k]['record_count']+1
        groups[k]['error_sum']=groups[k]['error_sum']+float(arr[2])
for k,v in groups.items:
    print ('{group}: {error_rate}'.format(group=k,error_rate=v['error_sum']/v['record_count']))

This code snippet stores all the groups in a dictionary, and calculates the error rate after reading the entire file.

It will encounter an out-of-memory exception, if there are too many combinations of groups.

Uri Goren
  • 13,386
  • 6
  • 58
  • 110
1

Q: Anyone knows what is happening?

A: Yes. Sum of all data memory-overheads for in-RAM objects !< RAM

It is a natural part of any formal abstraction to add some additional overhead in case some additional features are to be implemented on a higher ( a more abstract ) layer. That means that the more abstract / the more feature-rich representation of any dataset was chosen, the more memory- & processing-overheads are to be expected.

ITEMasINT             =                                32345
ITEMasTUPLE           =                              ( 32345, )
ITEMasLIST            =                              [ 32345, ]
ITEMasARRAY           = np.array(                    [ 32345, ] )
ITEMasDICT            =                         { 0:   32345, }


######## .__sizeof__()             -> int\nsize of object in memory, in bytes'
ITEMasINT.__sizeof__()             ->   12 #_____ 100% _ trivial INT
ITEMasTUPLE.__sizeof__()           ->   16 #      133% _ en-tuple-d
ITEMasLIST.__sizeof__()            ->   24 #      200% _ list-ed
ITEMasARRAY.__sizeof__()           ->   40 #      333% _ numpy-wrapped
ITEMasDICT.__sizeof__()            ->  124 #     1033% _ hash-associated asDict

If a personal experience is not enough, check the "costs" of re-wrapping the input ( already not small ) data into pandas overheads:

CParserError: Error tokenizing data. C error: out of memory
Segmentation fault (core dumped)

and

CParserError: Error tokenizing data. C error: out of memory
*** glibc detected *** python: free(): ...
...
..
.
Aborted (core dumped)

Q: Maybe there will be a solution?

A: Yes.

Simply follow the computational strategy and deploy memory-efficient & fast processing of the csv-input ( it's still a fileIO having some 8-15 ms access time and quite a low performance stream data-flow, even if you use SSD-devices with about 960MB/s peak-transfer rate, your blocking-fact is the memory-allocation limit ... so rather be patient on input-stream and do not crash into a principal memory-barrier for any in-RAM super-object ( which would have been introduced just to be finally asked ( if it did not crash during it's instantiation ... ) to compute a plain sum/nROWs ).

A line-by-line or block-arranged reads allow you to calculate results on-the-fly and using a register-based ( asDict and alike for an interim storage of results ) sliding-window computation strategy is both fast and memory-efficient. ( Uri has provided an example for such )

This principal approach is used to be used in both real-time constrained systems and for system-on-chip designs, that were used for processing large data-streams for more than the last half century, so nothing new uder the Sun.

In case your results's size cannot fit in RAM, than it makes no sense to even start the processing of any input file, does it?

Processing BigData is neither about super-up-scaling of the COTS-dataObjects nor about finding a best or a most sexy "one-liner" ...

BigData requires a lot of understanding of the way how to process both fast and smart so as to avoid extreme costs of even small overheads, that are forgiving to do principal mistakes on just a few GB-s of small-bigData but will kill anyone's budget & efforts once trying the same on a larger playground.

Community
  • 1
  • 1
user3666197
  • 1
  • 6
  • 50
  • 92