5

When I use Pymysql to connect to a Mysql database and create SScursor to query a stored procedure that returns about 600,000 pieces of data and processes that data every time it returns a day, the code is as follows:

def get_citatorlni_gen(self):
con = pymysql.connect(user=self.user, password=self.password, database=self.database,
                      host=self.host, port=self.port)
cur = con.cursor(pymysql.cursors.SSCursor)
try:
    cur.callproc('users_balance')
    while True:
        row = cur.read_next()
        if not row:
            break
        yield row[0]

except Exception:
    raise
finally:
    cur.close()
    con.close()

The processing stored procedure returns the following data:

def chunk_gen(self):
user_balance = self.users_balance()
for count, item in enumerate(user_balance, 1):
    if count % 100 == 0:
        time.sleep(5)  # Each batch takes about 5 seconds to process
else:
    time.sleep(5)

After about 10 minutes at a time, the following error occurs

  File "/user/src/app/controller.py", line 54, in users_balance
    row = cur.read_next()
  File "/home/.local/lib/python3.6/site-packages/pymysql/cursors.py", line 464, in read_next
    return self._conv_row(self._result._read_rowdata_packet_unbuffered())
  File "/home/.local/lib/python3.6/site-packages/pymysql/connections.py", line 1151, in _read_rowdata_packet_unbuffered
    packet = self.connection._read_packet()
  File "/home/.local/lib/python3.6/site-packages/pymysql/connections.py", line 673, in _read_packet
    recv_data = self._read_bytes(bytes_to_read)
  File "/home/.local/lib/python3.6/site-packages/pymysql/connections.py", line 698, in _read_bytes
    "Lost connection to MySQL server during query (%s)" % (e,))
pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([Errno 104] Connection reset by peer)')

Database side timeout configuration is as follows:

delayed_insert_timeout    300
have_statement_timeout    YES
innodb_flush_log_at_timeout    1
innodb_lock_wait_timeout    50
innodb_rollback_on_timeout    OFF
interactive_timeout    28800
lock_wait_timeout    31536000
net_read_timeout    30
net_write_timeout    60
rpl_stop_slave_timeout    31536000
slave_net_timeout    60
wait_timeout    28800
burningLk
  • 51
  • 2

0 Answers0