When I use Pymysql to connect to a Mysql database and create SScursor to query a stored procedure that returns about 600,000 pieces of data and processes that data every time it returns a day, the code is as follows:
def get_citatorlni_gen(self):
con = pymysql.connect(user=self.user, password=self.password, database=self.database,
host=self.host, port=self.port)
cur = con.cursor(pymysql.cursors.SSCursor)
try:
cur.callproc('users_balance')
while True:
row = cur.read_next()
if not row:
break
yield row[0]
except Exception:
raise
finally:
cur.close()
con.close()
The processing stored procedure returns the following data:
def chunk_gen(self):
user_balance = self.users_balance()
for count, item in enumerate(user_balance, 1):
if count % 100 == 0:
time.sleep(5) # Each batch takes about 5 seconds to process
else:
time.sleep(5)
After about 10 minutes at a time, the following error occurs
File "/user/src/app/controller.py", line 54, in users_balance
row = cur.read_next()
File "/home/.local/lib/python3.6/site-packages/pymysql/cursors.py", line 464, in read_next
return self._conv_row(self._result._read_rowdata_packet_unbuffered())
File "/home/.local/lib/python3.6/site-packages/pymysql/connections.py", line 1151, in _read_rowdata_packet_unbuffered
packet = self.connection._read_packet()
File "/home/.local/lib/python3.6/site-packages/pymysql/connections.py", line 673, in _read_packet
recv_data = self._read_bytes(bytes_to_read)
File "/home/.local/lib/python3.6/site-packages/pymysql/connections.py", line 698, in _read_bytes
"Lost connection to MySQL server during query (%s)" % (e,))
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([Errno 104] Connection reset by peer)')
Database side timeout configuration is as follows:
delayed_insert_timeout 300
have_statement_timeout YES
innodb_flush_log_at_timeout 1
innodb_lock_wait_timeout 50
innodb_rollback_on_timeout OFF
interactive_timeout 28800
lock_wait_timeout 31536000
net_read_timeout 30
net_write_timeout 60
rpl_stop_slave_timeout 31536000
slave_net_timeout 60
wait_timeout 28800