0

Good Day,

Today I got this problem on our database replica.

Read Replica Replication Error - IOError: 1236, reason: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from position > file size; the first event 'mysql-bin-changelog.305805' at 61005, the last event read from '/rdsdbdata/log/binlog/mysql-bin-changelog.305805' at 4, the last byte read from '/rdsdbdata/log/binlog/mysql-bin-changelog.305805' at 4.'

Can anyone point out what happen?

Thank you in advance.

PravinS
  • 2,640
  • 3
  • 21
  • 25
Henry
  • 1
  • 1
  • That sounds like a crash occurred on the master, resulting in the binlog being either trimmed during crash recovery or the log being otherwise lost entirely. 61005 is a byte offset -- so, not a lot of data, could potentially just be one transaction. There is a significant amount of forensic work I would be doing on these servers if they were mine, but it is rather too complex to describe in brief, and RDS does not allow access to all the necessary files (like the relay logs on the replica). Are you aware of a recent crash on the master? Is the master Multi-AZ? – Michael - sqlbot Mar 29 '17 at 02:35
  • hi Michael, actually i don't have access to our RDS. and I'm not aware of the crash on the master. Yes, the master is Multi-AZ. is it possible that the error came from mysql queries? thanks for the reply. – Henry Mar 29 '17 at 03:04
  • This issue should not have been caused *directly* by a query, but could have been a side-effect of a query that caused a crash by triggering a bug or by exposing latent corruption. The RDS instances -- both master and replica -- have logs accessible in the AWS console and accessing those (quickly, since they will eventually be purged) will be critical to understanding what has happened. – Michael - sqlbot Mar 29 '17 at 09:41
  • thank you very much @Michael-sqlbot . – Henry Mar 30 '17 at 05:16
  • I apologize that it doesn't seem practical to diagnose an issue like this, in this forum. Something has happened here that that would be essentially impossible on a system that hasn't experienced *some* kind of failure. It's more difficult to diagnose on RDS than on a standalone MySQL server, since the "managed" aspect of RDS prevents access to some of the potentially useful information, and a failover event in RDS Multi-AZ hides the failed machine from you. You should, however, be able to determine that a failover did occur, from the console. Consider contacting AWS support or a consultant. – Michael - sqlbot Mar 30 '17 at 14:38

0 Answers0