Get email senders in batch from Gmail API for over 10k emails

Question

I'm making an Chrome Extension which interacts with Gmail API. Basically it needs to extract email senders from emails that match some keyword. Problem is that sometimes there are over 10K emails matching this keyword. I utilize gapi.client.gmail.users.messages.list to fetch all emails that match entered keyword, but this only returns email and thread IDs, so I need to call gapi.client.gmail.users.messages.get for each email ID retrieved from messages.list. So there are over 10K requests to Gmail API, and I run into ERR_INSUFFICIENT_RESOURCES error in Chrome. To prevent this error, I set some timeout between calls to messages.get but then it would take ages to complete...

Is there some recommended way to get this amount of emails from Gmail API?

Iván Nokonoko · Accepted Answer · 2018-03-16T11:40:19.187

1

According to documentation, one way to improve performance is by batching requests. Currently it has a limit of 100 requests per batch, but it is still 100 times fewer requests.

EDIT: Also you can use fields parameter in query to request the fields you want from the messages, since both messages.list and messages.get can return a whole users.messages Resource.

For example:

var xhr = new XMLHttpRequest;
xhr.onload = () => {
    var resp = JSON.parse(xhr.response);
    var wholeFirstMessage = atob(resp.messages[0].raw);
    console.log(wholeFirstMessage);
};
xhr.open("GET", "https://www.googleapis.com/gmail/v1/users/userId/messages?fields=messages(id,threadId,raw)");
xhr.send();

NOTE: The code sample above ignores pageToken and maxResults from the XHR parameters for simplicity' sake. Those parameters would be needed for a long list of messages.

edited Mar 16 '18 at 11:40

answered Mar 16 '18 at 08:15

Iván Nokonoko

4,888
2
19
27

Thanks, I read about batch and implemented it successfully! Regarding `fields` parameter - I only need `From` from `payload > headers`. So can it be entered in `fields` parameter in some way (as it's nested property)? – kecman Mar 16 '18 at 12:49
I've added `'format': 'metadata'` to the call to `get` endpoint, but it would be great if I could get `From` header right away from `list` endpoint, instead of email IDs – kecman Mar 16 '18 at 12:57
@kecman You cannot get a specific header, but you can retrieve all headers directly from the `list` endpoint, with: `xhr.open("GET","https://www.googleapis.com/gmail/v1/users/userId/messages?fields=messages/payload/headers/*");`. More info [here](https://developers.google.com/gmail/api/guides/performance#partial-response). – Iván Nokonoko Mar 16 '18 at 15:36
I'm using JavaScript library. When I provide it with `fields` parameter like this `messages/payload/headers/*` it returns empty 200 OK response. I found this comment regarding it, do you have any other idea? https://stackoverflow.com/questions/25484791/gmail-api-users-messages-list#comment81074139_25494460 – kecman Mar 16 '18 at 19:30
1

@kecman I'm afraid I hadn't tested it myself, I just followed the documentation. There seems to be a bug in the API. No other idea than batching `list` requests and `get` requests afterwards. – Iván Nokonoko Mar 16 '18 at 20:08
Thanks, it helps in some way, I will accept this answer! – kecman Mar 16 '18 at 21:50

score 0 · Answer 2 · answered Mar 16 '18 at 07:08

This is unfortunately how the Gmail api works.

You are going to have to do a messages.list followed by a message.get if you want to get more details about the message.

There is a limit to how fast you can make requests against the api you just need to slow things down if you get the error. Flooding error messages are used to ensure that we all get to use the api and one person doesnt over load things by making to many requests.

Get email senders in batch from Gmail API for over 10k emails

2 Answers2