Most efficient way to get new messages

Question

I have a process that looks for new messages in a users Gmail. The message is added to an external database if it meets certain address criteria.

We have been using Users.History.List which returns all messages that have had a change made to them. This is quite inefficient as we have to subsequently check each message to see if we have already processed it.

We are looking at alternatively using Users.Messages.List and checking the MsgId to see if it is larger than the previous check (we store the Id from that). The assumption here is that the MsgId will keep getting larger. Is this approach flawed? What are others doing?

Many Thanks.

I'm facing the same situation, it works but as you say, it's inefficient. Did you figure out a better way to deal with this? — gdvalderrama, Nov 10 '17 at 15:21

score 6 · Answer 1 · answered Apr 07 '15 at 10:50

Here is an example how to get only new message (using Java client api)

List<History> histories = new ArrayList<History>();
ListHistoryResponse response = service.users().history().list(userUID).setStartHistoryId(startHistoryId).execute();

//get all history events and not only the first page
while (response.getHistory() != null) {
    histories.addAll(response.getHistory());
    if (response.getNextPageToken() != null) {
        String pageToken = response.getNextPageToken();
        response = service.users().history().list(userUID)
                .setPageToken(pageToken)
                .setStartHistoryId(startHistoryId).execute();
    } else {
        break;
    }
}

//for each history element find the added messages
for (History history : histories) {
    List<HistoryMessageAdded> addedMessages = history.getMessagesAdded();

    if (addedMessages == null){
        continue;
    }

    //call to message.get for each HistoryMessageAdded
    for (HistoryMessageAdded addedMsg : addedMessages) {
        Message message = addedMsg.getMessage();
            Message rawMessage = service.users().messages().get(userUID, message.getId()).setFormat("raw").execute();

    }
}

Probably there is a similar implementation in other languages/REST API.

You can use other history events such as:messagesDeleted, labelsAdded and labelsRemoved Reference: https://developers.google.com/gmail/api/v1/reference/users/history/list

score 3 · Answer 2 · edited Feb 25 '15 at 19:43

3

Message ID is unique and its value is never changed. To get new messages, you can use history.list(), and give historyId of whatever the largest historyId you've previously for the message.

Here is the example response:

{
 "history": [
  {
   "id": "1825077",
   "messages": [
    {
     "id": "14b4c0dbc6ba9a57",
     "threadId": "14b4b4ae8cfbea5c"
    }
   ]
  },
  {
   "id": "1825087",
   "messages": [
    {
     "id": "14b4c0dc3ab5e49b",
     "threadId": "14b4b4ae8cfbea5c"
    }
   ]
  },
  {
   "id": "1825097",
   "messages": [
    {
     "id": "14b4c0e07e0f6545",
     "threadId": "14b4b4ae8cfbea5c"
    }
   ]
  }
 ]
}

1825097 is the largest historyId for the message "14b4c0e07e0f6545". Also Msgid didn't change here only history id is changed.

If you give 1825097 as history id and there is no change in the message, then the response will be 200 with headers. If you get response 404 error, you'll need to use messages.list() instead to perform a full sync.

edited Feb 25 '15 at 19:43

Rafa Viotti

9,998
4
42
62

answered Feb 02 '15 at 21:20

SGC

1,025
1
6
6

3

We have been using the History Id but this returns all changes to threads/messages. We only want new messages and want to avoid checking all the history against the messages we have already synchronised. We only want a list of new messages if possible. – PNC Feb 02 '15 at 23:37
You can use messages.list(), give the label name and q value as "is:unread" to get all the unread messages. – SGC Feb 02 '15 at 23:43
Yes we are messages.list() and get all messages where the id is larger than the last synchronised messageId. This seems to be the best way. Unread would ignore any sent messages/opened messages I would have thought? – PNC Feb 03 '15 at 00:58
@PNC is there any official documentation which suggest that messageIds are always in increasing order . ? – vinit payal Mar 27 '18 at 09:15
@vinitpayal - no official documentation but we have been using this approach for several years now without issue. – PNC Mar 27 '18 at 17:50
Great also @PNC how are you guys passing largest message id which you have already synced to `messages.list` api as looking into https://developers.google.com/gmail/api/v1/reference/users/messages/list there is no parameter for it. – vinit payal Mar 28 '18 at 06:06

Most efficient way to get new messages

2 Answers2

Linked