The code below is from the vaex documentation:
pandas_df = pd.read_sql_query('SELECT * FROM MYTABLE', con=engine)
df = vaex.from_pandas(pandas_df, copy_index=False)
Description
I have data more than RAM. But, when I use above code, it try and pull all data in panda dataframe. So to solve this I used chunksize attribute which gives a generator.
To convert from generator to pandas dataframe again it is needs memory. Below is the code I tried.
import vaex
df = pd.read_sql_query('select * from "user"."table"', conn, chunksize=1000000)
chunk_list = []
for i in df:
chunk_list.append(i)
data = pd.concat(chunk_list)
df2 = vaex.from_pandas(data)
alldat=df2.concat(df2)
Please help me with this issue.
