7

Say i have only 1GB of memory and 1 TB of hard disk space.

This is my code and i am using a postgres database. 

import psycopg2

try:
   db = psycopg2.connect("database parameters")
   conn = db.cursor()
   conn.execute(query) 

   #At this point, i am running 
   for row in conn:

for this case, I guess it is safe to assume that conn is a generator as i cannot seem to find a definitive answer online and i cannot try it on my environment as i cannot afford the system to crash.

I am expecting this query to return data in excess of 100 GB

I am using python 2.7 and psycopg2 library

aceminer
  • 4,089
  • 9
  • 56
  • 104
  • Why not write a query that returns only one result and try `(isinstance(gen, types.GeneratorType)` as per [this answer](http://stackoverflow.com/questions/6416538/how-to-check-if-an-object-is-a-generator-object-in-python)? That should tell you at once if `conn.execute()` returns a generator. – Akshat Mahajan Apr 05 '16 at 03:49

1 Answers1

11

If you use an anonymous cursor, which you are doing in your example, then the entire query result will be read into memory.

If you use a named cursor then it will read from the server in chunks as it loops over the data.

jjanes
  • 37,812
  • 5
  • 27
  • 34