How do I get pagination working with exclusiveStartKey for dynamodb aws-sdk-go-v2?

Question

I'm trying to create a pagination endpoint for a dynamodb table I have. But I've tried everything to get the exclusiveStartKey to be the correct type for it to work. However, everything I've tried doesn't seem to work.

example code:

func GetPaginator(tableName string, limit int32, lastEvaluatedKey string) (*dynamodb.ScanPaginator, error) {
    svc, err := GetClient()
    if err != nil {
        logrus.Error(err)
        return nil, err
    }

    input := &dynamodb.ScanInput{
        TableName: aws.String(tableName),
        Limit:     aws.Int32(limit),
    }

    if lastEvaluatedKey != "" {
        input.ExclusiveStartKey = map[string]types.AttributeValue{
            "id": &types.AttributeValueMemberS{
                Value: lastEvaluatedKey,
            },
        }
    }

    paginator := dynamodb.NewScanPaginator(svc, input)
    return paginator, nil
}

Edit:

Okay so I'm creating a API that requires pagination. The API needs to have a query parameter where the lastEvaluatedId can be defined. I can then use the lastEvaluatedId to pass as the ExclusiveStartKey on the ScanInput. However when I do this I still received the same item from the database. I've created a test.go file and will post the code below:

package main

import (
    "context"
    "fmt"
    "os"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
    "github.com/aws/aws-sdk-go-v2/service/dynamodb"
)

type PaginateID struct {
    ID string `dynamodbav:"id" json:"id"`
}

func main() {
    lastKey := PaginateID{ID: "ae82a99d-486e-11ec-a7a7-0242ac110002"}

    key, err := attributevalue.MarshalMap(lastKey)
    if err != nil {
        fmt.Println(err)
        return
    }

    cfg, err := config.LoadDefaultConfig(context.TODO(), func(o *config.LoadOptions) error {
        o.Region = os.Getenv("REGION")
        return nil
    })
    if err != nil {
        fmt.Println(err)
        return
    }

    svc := dynamodb.NewFromConfig(cfg, func(o *dynamodb.Options) {
        o.EndpointResolver = dynamodb.EndpointResolverFromURL("http://localhost:8000")
    })

    input := &dynamodb.ScanInput{
        TableName:         aws.String("TABLE_NAME"),
        Limit:             aws.Int32(1),
        ExclusiveStartKey: key,
    }

    paginator := dynamodb.NewScanPaginator(svc, input)

    if paginator.HasMorePages() {
        data, err := paginator.NextPage(context.TODO())
        if err != nil {
            fmt.Println(err)
            return
        }

        fmt.Println(data.Items[0]["id"])
        fmt.Println(data.LastEvaluatedKey["id"])
    }
}

When I run this test code. I get this output:

&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}
&{ae82a99d-486e-11ec-a7a7-0242ac110002 {}}

So the item that is returned is the same Id that I am passing to the ScanInput.ExclusiveStartKey. Which means it's not starting from the ExclusiveStartKey. The scan is starting from the beginning everytime.

Answer 1

so basically what you need to do is to get the LastEvaluatedKey and to pass it to ExclusiveStartKey

you can not use the scan paginator attributes because it's not exported attributes, therefore instead I suggest that you use the returned page by calling NextPage

in the following snippet I have an example :

func GetPaginator(ctx context.Context,tableName string, limit int32, lastEvaluatedKey map[string]types.AttributeValue) (*dynamodb.ScanOutput, error) {
    svc, err := GetClient()
    if err != nil {
        logrus.Error(err)
        return nil, err
    }

    input := &dynamodb.ScanInput{
        TableName: aws.String(tableName),
        Limit:     aws.Int32(limit),
    }

    if len(lastEvaluatedKey) > 0  {
        input.ExclusiveStartKey = lastEvaluatedKey
    }

    paginator := dynamodb.NewScanPaginator(svc, input)
    
    return paginator.NextPage(), nil
}

keep in mind that paginator.NextPage(ctx) could be nil incase there is no more pages or you can use HasMorePages()

Answer 2

The aws-sdk-go-v2 DynamoDB query and scan paginator constructors have a bug (see my github issue, includes the fix). They do not respect the ExclusiveStartKey param.

As an interim fix, I copied the paginator type locally and added one line in to the constructor: nextToken: params.ExclusiveStartKey.