0

So I have a huge data frame that I am writing to a table in a SQL database by splitting the data frame into chunks of size 10,000 (I was told that speeds up the process). I use the following code:

   from sqlalchemy import create_engine

   my_username = 'John_Doe'
   my_password = 'fake_password'
   my_hostname = 'something_long'
   my_database = 'generic_db_name'
   my_schema = 'i_forget'

   my_engine = create_engine('postgresql://'+ my_username + ':' + my_password + '@' + my_hostname + 
   ':' +'5432/' + my_database)

   my_df.to_sql('this_is_what_im_naming_the_tbl', con = my_engine, schema='my_schema', if_exists = 'replace', 
   index = False, chunksize = 10000)

Would there be a way to print a message whenever a new chunk gets written into the table? Maybe something like "a chunk was just added to the table". It's a postgresql database but that information probably doesn't help.

Vincent_Adultman
  • 78
  • 11
  • if you use python-postgresql client like Psycopg to do the database execution,you can return the generated ID back after inserting the row. – Rajith Thennakoon Oct 03 '19 at 05:16
  • @Rajith_Thennakoon I'm familiar with psycopg for reading from sql tables but not for writing into tables. Do you happen to know the code that needs to be executed? – Vincent_Adultman Oct 03 '19 at 10:23

0 Answers0