How do you use "NextToken" in AWS API calls

Question

I've run into a little issue that I am really struggling to understand how it works. I have a tool I am writing that basically does a describe-organization to collect all the accounts in our AWS organization. Per the documentation here it says it responds with a json of the accounts which in my case will be hundreds and hundreds of accounts. So I wrote some very simple code to switch roles into our master account and make the call:

import boto3
import uuid
import pprint

iam_client = boto3.client('iam')
sts_client = boto3.client('sts')
org_client = boto3.client('organizations')


print("Starting in account: %s" % sts_client.get_caller_identity().get('Account'))

assumedRoleObject = sts_client.assume_role(
    RoleArn="arn:aws:iam::123456xxx:role/MsCrossAccountAccessRole",
    RoleSessionName="MasterPayer"
)

credentials = assumedRoleObject['Credentials']

org_client = boto3.client(
    'organizations',
    aws_access_key_id = credentials['AccessKeyId'],
    aws_secret_access_key = credentials['SecretAccessKey'],
    aws_session_token = credentials['SessionToken'],
)

getListAccounts = org_client.list_accounts(
    NextToken='string'
)

But when I execute the code, I get the following error:

"botocore.errorfactory.InvalidInputException: An error occurred (InvalidInputException) when calling the ListAccounts operation: You specified an invalid value for nextToken. You must get the value from the response to a previous call to the API."

I'm really stumped on what that means. I see the NextToken, and I can find many references to it in the AWS documentation but I can't figure out how to actually USE it. Like, what do I need to do with it?

Let me suggest using this lib: https://github.com/jiss2891/awstanding — Juan Ignacio Sánchez, Oct 08 '21 at 01:20

score 34 · Accepted Answer · answered Aug 29 '18 at 02:02

Don't take the boto3 examples literally (they are not actual examples). Here is how this works:

1) The first time you make a call to list_accounts you'll do it without the NextToken, so simply

getListAccounts = org_client.list_accounts()

2) This will return a JSON response which looks roughly like this (this is what is saved in your getListAccounts variable):

{
    "Accounts": [<lots of accounts information>], 
    "NextToken": <some token>
}

Note that the NextToken is only returned in case you have more accounts than one list_accounts call can return, usually this is 100 (the boto3 documentation does not state how many by default). If all accounts were returned in one call there is no NextToken in the response!

3) So if and only if not all accounts were returned in the first call you now want to return more accounts and you will have to use the NextToken in order to do this:

getListAccountsMore = org_client.list_accounts(NextToken=getListAccounts['NextToken'])

4) Repeat until no NextToken is returned in the response anymore (then you retrieved all accounts).

This is how the AWS SDK handles pagination in many cases. You will see the usage of the NextToken in other service clients as well.

Thank you that is perfect. Yeah the documentation seems to suggest that NextToken is mandatory and it was driving me nuts. — Geoff Sweet, Aug 29 '18 at 18:35

score 32 · Answer 2 · answered Nov 05 '20 at 13:30

32

Same as other answer, but with a short snippet with a simple while loop.

response = client.list_accounts()
results = response["Accounts"]
while "NextToken" in response:
    response = client.list_accounts(NextToken=response["NextToken"])
    results.extend(response["Accounts"])

answered Nov 05 '20 at 13:30

Vincent J

4,968
4
40
50

Thanks vincent , it is easier to follow urs and able to achieve list of queues client = boto3.client('sqs') queuenames = client.list_queues(MaxResults=1000) queue_url = queuenames['QueueUrls'] while "NextToken" in queuenames: queuenames = client.list_queues(MaxResults=1000,NextToken=queuenames["NextToken"]) queue_url.extend(queuenames['QueueUrls']) – karthik Nov 20 '20 at 22:13

score 7 · Answer 3 · answered Aug 21 '20 at 13:39

Instead you can use get_paginator api. find below example, In my use case i had to get all the values of SSM parameter store and wanted to compare it with a string.

import boto3
import sys

LBURL = sys.argv[1].strip()
client = boto3.client('ssm')
p = client.get_paginator('describe_parameters')
paginator = p.paginate().build_full_result()
for page in paginator['Parameters']:
    response = client.get_parameter(Name=page['Name'])
    value = response['Parameter']['Value']
    if LBURL in value:
        print("Name is: " + page['Name'] + " and Value is: " + value)

This solution is way better - much simpler login flow. – lapkritinis Sep 13 '20 at 11:41 — lapkritinis, Sep 13 '20 at 11:41

score 2 · Answer 4 · answered Feb 11 '20 at 09:20

Here my example where I used NextToken to check if a secret exist inside the SecretManager. There are also some print useful to visualize the first time

def check_if_secret_existv2(username):
    results_for_call=5
    response = client.list_secrets(MaxResults=results_for_call)
    i=0
    while True:
        i=i+1
        if 'NextToken' in response:
            response = client.list_secrets(MaxResults=results_for_call,NextToken=response['NextToken'])
        else:
            response = client.list_secrets(MaxResults=results_for_call)

        for secret in response['SecretList']:
            print(secret['Name'])
            if secret['Name'] == username:
                return True
        print('End cycle '+str(i))

        if 'NextToken' not in response:
            break
    return False

print(check_if_secret_existv2(myusername))

I am thinking if you don't have logical error in this flow - you are not outputting first 5 results, are you? — lapkritinis, Sep 13 '20 at 11:34

score 2 · Answer 5 · answered Aug 25 '20 at 19:48

I know it's a different client but for anyone still having issues this took me some time to figure out:

This client required the initial request params also be included. Here's the code:

import boto3

lex_client = boto3.client("lex-models")
response = lex_client.get_slot_types(
    nameContains='search_string',
    maxResults=50
)
slot_types = response['slotTypes']
while 'nextToken' in response.keys():
    response = lex_client.get_slot_types(nameContains='search_string', maxResults=50, nextToken=response['nextToken'])
    slot_types.extend(response['slotTypes'])
print('here are the slot types', slot_types)

score 2 · Answer 6 · edited Jun 25 '21 at 19:28

I tried to list secrets names in my secrets manager using boto3 python:

secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])

for secret in secrets_manager: 
    print ("{0}".format(secret['Name']))

The complete list was around 20, but the output was only 5 secret names.

Updated the code to below, it worked:

secrets = secret_client.list_secrets()
secrets_manager = (secrets['SecretList'])

while "NextToken" in secrets:
    secrets = secret_client.list_secrets(NextToken=secrets["NextToken"])
    secrets_manager.extend(secrets['SecretList'])

for secret in secrets_manager: 
    print ("{0}".format(secret['Name']))

score 0 · Answer 7 · answered Aug 09 '22 at 01:15

And another example to return a list of LogGroupNames using nextToken:

    response = logs_client.describe_log_groups(limit=50)
    results = response["logGroups"]
    log.info(f'    response: {response}')
    log.info(f'    results: {results}')
    while "nextToken" in response:
        log.info('    nextToken')
        response = logs_client.describe_log_groups(nextToken=response["nextToken"], limit=50)
        results.extend(response["logGroups"])

    log_group_name_list = []
    for groups in results:
        log_group_name = groups['logGroupName']
        log_group_name_list.append(log_group_name)
    log.info(f'    log_group_name_list: {log_group_name_list}')
    log.info(f'    log_group_name_list:type: {type(log_group_name_list)}')

score 0 · Answer 8 · answered Mar 09 '23 at 21:26

0

resp = dynamodb.execute_statement(Statement="SELECT gateway FROM 
Table_Name")
data = resp['Items']
#print(resp['NextToken'])
while 'NextToken' in resp:
    resp = dynamodb.execute_statement(Statement="SELECT gateway FROM 
    Table_Name", NextToken =resp['NextToken'])
    data.extend(resp['Items'])

answered Mar 09 '23 at 21:26

mario42004

1

I have used the code above to perform a pagination using PartiSQL statement. – mario42004 Mar 09 '23 at 21:27
As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Mar 12 '23 at 15:36

How do you use "NextToken" in AWS API calls

8 Answers8

Linked