0

We know from the documentation there is a theoretical limit of 1 message per user per second, but we aren't coming anywhere close to that while running email migrations on a high-end server. What should we do? Should we increase the amount of threads per user to more than one (even though the documentation suggests only 1 thread per user)? I've used their GAMME tool and it blows the email migration api away in terms of speed, even on lower end servers.

Does anyone have any suggestions? It's not super-slow, but it's slow enough to be a pain.

roger34
  • 275
  • 2
  • 6
  • 16

1 Answers1

1

The GAMME tool itself utilizes the Email Migration API, it's not doing anything special so there are likely other factors slowing your migration. Are you actually hitting the migration API from AppEngine? If so, you should be able to utilize appstats to profile your application and see if there are other bottlenecks. Where are you pulling messages from?

Do not attempt to use more than 1 thread per user migration, it won't work and you'll get performance issues. DO make sure that you are properly implementing exponential backoff. If your app doesn't acknowledge 503 error codes by backing off exponential (1 second the first time, then 2 seconds, 4, 8, etc) then Google will respond by further throttling your API calls.

Jay Lee
  • 13,415
  • 3
  • 28
  • 59
  • Thanks for answering! With the exponential backoffs, how many times do you have it backoff before saying, okay it's not going to move, go onto the next one? – roger34 Jan 19 '13 at 01:04
  • 1
    I believe GAMME defaults to 5 trys. I usually do 10 trys but I max out the backoff at 60 seconds. So I was 1, 2, 4, 8, 16, 32, 60, 60, 60, 60. Also note that the Drive APIs exponential backoff docs also recommend you add a random number of milliseconds (0 to 1000) to the backoff time in order to "avoid certain lock errors in some concurrent implementations". NO idea if that applies to Mail Migration or not but waiting an additional second or less can't hurt. https://developers.google.com/drive/handle-errors#implementing_exponential_backoff – Jay Lee Jan 19 '13 at 01:11
  • Sounds like our backoffs are similar...What other factors could be causing the slowness? We're running the migrations on large AWS servers so the bandwidth/resources shouldn't be a problem. – roger34 Jan 19 '13 at 01:22
  • how large are the messages? Bandwidth can obviously cause slowdowns. Make sure you're bottleneck isn't bandwidth or I/O. It could be a bottleneck between your app and the source mail datastore or it could be between AWS and Google's servers. – Jay Lee Jan 26 '13 at 14:19