2

The problem: We have several lambdas and dynamodb tables we are using in production, when releasing a new version of our code we sometimes strip an attribute or add attributes to our table classes (Java code using com.amazonaws.services.dynamodbv2.datamodeling) High level api. When we deploy the new version of the code and we query the table, if an new attribute does not exist for an existing item, or we remove an attribute from the code. It breaks the code because our Item object is out of line with the production data.

We would like to avoid treating data in prod by adding extra attributes with a default value or removing attributes to existing items. Before we deploy the new version for a variety of reasons concerning race conditions and consistency. What would be preferable if we handled it at the code level, if the attribute does not exist automatically add it with a default value. Or have the code ignore attributes that are not defined in the item/table class. Is this possible using the high-level java sdk api?

The other solution we came up with was to create a service that is fed the delta (change between the code item object and data in prod), that is executed by a pretraffic lambda that treats the data when deploying a new version of the lambda. We would like to avoid this however.

package com.ourcompany.module.dynamodb.items;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBAttribute;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBVersionAttribute;
import lombok.Data;

@Data
@DynamoDBTable(tableName = "Boxes")
public class BoxItem {

@DynamoDBHashKey(attributeName = "boxID")
private String channelID;

#This is the field we added, the previous version did not have this field, in prod we have many items without this attribute
@DynamoDBAttribute(attributeName = "lastTimeAccess")
private String lastTimeAccess;

@DynamoDBAttribute(attributeName = "initTime")
private String initTime;

@DynamoDBAttribute(attributeName = "boxIDhash")
private String streamBoxIDHash;

@DynamoDBAttribute(attributeName = "CFD")
private String cfd;

@DynamoDBAttribute(attributeName = "originDomain")
private String originDomain;

@DynamoDBAttribute(attributeName = "lIP")
private String lIP;

@DynamoDBAttribute(attributeName = "pDomain")
private String pDomain;

Above is our item that was changed, with added attribute.

package com.ourcompany.shared.module.repository.dynamob;

import ...

public class DynamoDbRepository<Item, Key> {

private final DynamoDBMapper mapper;
private static final Logger logger = LogManager.getLogger(DynamoDbRepository.class);

@Inject
public DynamoDbRepository() {
    val client = AmazonDynamoDBClientBuilder
            .standard()
            .withRegion(Regions.US_EAST_1) // TODO: hardcoded now
            .withRequestHandlers(new TracingHandler(AWSXRay.getGlobalRecorder()))
            .build();


    DynamoDBMapperConfig dynamoDBMapperConfig = new DynamoDBMapperConfig.Builder()
                                                   .withSaveBehavior(DynamoDBMapperConfig.SaveBehavior.UPDATE_SKIP_NULL_ATTRIBUTES)
                                                   .withTableNameResolver(new DynamoDBTableNameResolver())
                                                   .build();

    mapper = new DynamoDBMapper(client, dynamoDBMapperConfig);

}
/*
* Many accessor methods are listed here below is the one where we have issue,
*/
public List<Item> findBy(Map<String, Condition> filter, final Class<Item> clazz) throws Exception {
    try {
        logger.trace("DynamoDbRepository findBy(filter, class)");
        val scanExpression = new DynamoDBScanExpression().withScanFilter(filter).withConsistentRead(true);
        PaginatedScanList<Item> ls = mapper.scan(clazz, scanExpression);
        ls.loadAllResults();
        return ls;
    } catch (Exception ex) {
        logger.trace(ex.getMessage());
        throw handleException(ex);
    }
}

Above is our Dynamob DB mapper class, but with only the method in question. We were able to trace through logging up to the line logger.trace("DynamoDbRepository findBy(filter, class)");, and we know the issue occurs in the mapper. However it does not spit the exception up, so we are not able to see the actual error. We had to solve the issue by purging all the data from the tables in prod then have the new version of code repopulate the entries with the attribute and the code worked.

Shravan Deolalikar
  • 233
  • 5
  • 15
  • DynamoDB has no schema that cares about them and attributes that don't exist are simply `null` when you query them. Why does it break for you? It shouldn't unless maybe you use primitives like `int` that can't be `null`. The usual way to go is to fix things while loading old entries, like add state that is expected to be there. – zapl Aug 27 '18 at 17:34
  • @zapl I understand there is no schema, as it is nosql. However the Java SDK fr DynamoDB does care. it breaks the code because it tries to resolve the item in the table as an object in the java code, as the tables are managed by the DynamoDBMapper. – Shravan Deolalikar Aug 27 '18 at 18:21
  • Well it shouldn't break with DynamoDBMapper, that class is intended to be able to handle attributes that don't exist or are new. By default it usually doesn't touch attributes you don't model in your class but you can also control how it behaves wrt to those by changing the SaveBehavior (see e.g. https://aws.amazon.com/blogs/developer/using-the-savebehavior-configuration-for-the-dynamodbmapper/). The only Issue I can see is using primitive types or attributes that aren't compatible pre/post migration. You should in that case design your attribute changes in a migration friendly way. – zapl Aug 27 '18 at 18:31
  • The problem is when reading data from the table using scan. The DynamoDBMapper enables you to alter the save behavior and set the ConversionSchema (But this has to do with the old supported version of DynamoDB data types). It tries to marshall the item from the db into the object and breaks. @zapl – Shravan Deolalikar Aug 29 '18 at 12:04
  • Can you add a concrete example to your question? Something that shows the crash, the reason it crashes, the change in attributes? It shouldn't be a problem to add / remove attributes, I suppose even change an attribute storage type (from S to N for example) if you handle that with a `DynamoDBTypeConverter`. It's fairly customizable when it comes to adjusting the marshalling. There are ways to change attributes that enable easier migration and there are changes that are hard to deal with. – zapl Aug 29 '18 at 18:11
  • @zapl I added some code and explanation. I hope you can shed some light =) – Shravan Deolalikar Aug 30 '18 at 13:08
  • The code looks almost exactly like I did several times successfully. Don't see anything wrong, except that you shouldn't log just `ex.getMessage()` because that almost always bites you in the end because there isn't always a message. At least use ex.toString() or better log the stacktrace with trace("whatever", ex) – zapl Sep 03 '18 at 08:51

2 Answers2

1

You will have this problem for a small window or if you run a long living split tests.

We solved with following ways:

  1. Whichever lambda is using the attributes make sure they check whether property exists and work on it. If a required property does not exist, throw an error and assume it failed. This can be a problem if you are using it in a transactional path, but will let you know what failed and how to fix it. This is for split tests.
  2. Architect your code for backward compatibility for atleast one version behind. Be sure to remove the code once the desired version is in place.
  3. If the window is small and not heavily loaded, you can let the service to fail to catch the newer version.

Hope it helps.

Kannaiyan
  • 12,554
  • 3
  • 44
  • 83
  • thanks for your response. Not totally clear about your recommendation, in point one you suggest checking the attribute whether it exists or not, I dont think this can be done using DynamoDB mapper. We would have to rewrite the whole data layer to leverage the low level api. @Kannaiyan – Shravan Deolalikar Aug 29 '18 at 12:07
  • It is not on the dynamoDB mapper. When any attribute is required by the business process, you need to make sure whether they exist or not and make your decisions. If some required property needed by one process got removed by another process it is going to break the system. You might need a monitor to indicate which process got affected and either fix the code or change that business process. – Kannaiyan Aug 29 '18 at 12:18
0

Just an update on the issue. After taking @zapl advice on trying to print the stacktrace, I found that there was absolutely nothing wrong with the way the AWS DynamoDB Mapper or SDK works. I was expecting to capture some stacktrace from the SDK, and did not, after some more careful tracing I discovered that the Java Devs misdiagnosed the issue, and the real issue is they had logic to filter a stream which depended on the new fields. So lesson of the story, architect the code for backward compatibility at least one version behind!

Shravan Deolalikar
  • 233
  • 5
  • 15