Importing database takes a lot of time

Question

I am trying to import a table that contains 81462 rows in a dataframe using the following code:

sql_conn = pyodbc.connect('DRIVER={SQL Server}; SERVER=server.database.windows.net; DATABASE=server_dev; uid=user; pwd=pw') 
query = "select * from product inner join brand on Product.BrandId = Brand.BrandId"
df = pd.read_sql(query, sql_conn)

And the whole process takes a very long time. I think that I am already 30-minutes in and it's still processing. I'd assume that this is not quite normal - so how else should I import it so the processing time is quicker?

try to import small part of data and check if connection is OK — Sociopath, Sep 16 '19 at 11:46
Already did. Works fine with 200, it imports almost instantly. — Questieme, Sep 16 '19 at 11:47
@VikasP even if I add `for chunk in pd.read_sql(query, sql_conn, chunksize=500): df=chunk` it still takes a lot of time. — Questieme, Sep 16 '19 at 11:57
@RomanPerekhrest I am not quite sure where that Fetch Next belongs, though. I will look into it — Questieme, Sep 16 '19 at 11:58

score 1 · Accepted Answer · answered Sep 16 '19 at 12:17

Thanks to @RomanPerekhrest. FETCH NEXT imported everything within 1-2 minutes.

SELECT product.Name, brand.Name as BrandName, description, size FROM Product inner join brand on product.brandid=brand.brandid ORDER BY Name OFFSET 1 ROWS FETCH NEXT 80000 ROWS ONLY

Importing database takes a lot of time

1 Answers1