AWS IOT ShadowManager - Synchronization error

Question

I'm working on an IoT project using AWS IoT and Greengrass v2 and I'm trying to integrate the ShadowManager component to use local shadows but when I deploy it on my device, it return a fatal exception during the synchronization step

{greengrass-root}/logs/greengrass.log

2021-09-15T09:54:29.044Z [INFO] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=LocalUpdateSyncRequest, thing name=mydevice, shadow name=}
2021-09-15T09:54:29.082Z [WARN] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Received conflict when processing request. Retrying as a full sync. {thing name=mydevice, shadow name=}
software.amazon.awssdk.aws.greengrass.model.ConflictError: Missed update(s) from the cloud
    at com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest.execute(LocalUpdateSyncRequest.java:142)
    at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
    at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
    at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
    at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

2021-09-15T09:54:29.083Z [INFO] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Executing sync request. {Type=FullShadowSyncRequest, thing name=mydevice, shadow name=}
2021-09-15T09:54:29.357Z [ERROR] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest: Could not execute cloud shadow get request. {thing name=mydevice, shadow name=}
2021-09-15T09:54:29.358Z [ERROR] (pool-2-thread-33) com.aws.greengrass.shadowmanager.sync.SyncHandler: sync. Skipping sync request. {thing name=mydevice, shadow name=}
com.aws.greengrass.shadowmanager.exception.SkipSyncRequestException: software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: null (Service: IotDataPlane, Status Code: 403, Request ID: 84d49520-0162-7416-61a4-9973ecd32dad, Extended Request ID: null)
    at com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest.getCloudShadowDocument(FullShadowSyncRequest.java:479)
    at com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest.execute(FullShadowSyncRequest.java:93)
    at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:136)
    at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49)
    at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:134)
    at com.aws.greengrass.shadowmanager.sync.SyncHandler.syncLoop(SyncHandler.java:270)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: software.amazon.awssdk.services.iotdataplane.model.IotDataPlaneException: null (Service: IotDataPlane, Status Code: 403, Request ID: 84d49520-0162-7416-61a4-9973ecd32dad, Extended Request ID: null)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
    at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
    at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
    at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
    at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
    at software.amazon.awssdk.services.iotdataplane.DefaultIotDataPlaneClient.getThingShadow(DefaultIotDataPlaneClient.java:221)
    at com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper.getThingShadow(IotDataPlaneClientWrapper.java:89)
    at com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest.getCloudShadowDocument(FullShadowSyncRequest.java:458)
    ... 10 more

It seems like the ShadowManager has not the good access rights but I checked multiple time and I have well added the good policies:

iot:GetThingShadow
iot:UpdateThingShadow
iot:DeleteThingShadow

I've tested with classic shadow and named shadow but same result.

The configuration of my ShadowManager is:

{
    "synchronize": {
        "coreThing": {
            "classic": true,
            "namedShadow": ["mydevice"]
        }
    }
}

It's also important to know that it worked the first time I've deployed it but after several minutes and some update messages published it failed (without changing anything).

Someone could help me on this?

Thank you

For better personalized support, you can submit a post on the [AWS IoT Greengrass Forums](http://forums.aws.amazon.com/forum.jspa?forumID=254) or create a customer support ticket. AWS IoT Greengrass engineers can interact with you on the forums to request more information such as your device's config and logs to dive deeper at the root problem. — Michael Dombrowski, Sep 27 '21 at 17:19

score 0 · Answer 1 · answered Sep 24 '21 at 15:28

I was having a similar issue but with a client device. This device was previously added to an old ggc v1. I migrated it to the ggc v2 but it seems that some config remained on the old server and it was overwriting my shadow. So while testing with the help of the MQTT test I would create a shadow, ggc v2 would detect a conflict and then it will do a full sync deleting the shadow I had sent. Using a completely new device avoided this error. I am still not able to sync sending the data from the client but at least from the MQTT test I can.

score 0 · Answer 2 · answered Sep 27 '21 at 17:19

It looks like you are getting a 403 during a full sync operation when it tries to get the cloud shadow from IoT Device Shadow service. This 403 indicates that the device doesn't have permission to get device shadows. These permissions are required on the core device's AWS IoT policy. To confirm the correct permissions, please see the documentation about the minimal AWS IoT policy for core devices.

score 0 · Answer 3 · answered Oct 20 '21 at 16:34

I was having the same issue and going crazy triple-checking configuration and IAM policies. Michael's answer will lead you to the right place, but the important thing to realize is that operations through the ShadowManager use AWS IoT Policies rather than IAM policies through the TokenExchangeRole.

From Device authentication and authorization for AWS IoT Greengrass

AWS IoT policies define the set of operations allowed for AWS IoT devices. Specifically, they allow and deny access to AWS IoT Core and AWS IoT Greengrass data plane operations, such as publishing MQTT messages and retrieving device shadows.

So, interactions with Device Shadows through the ShadowManager are IoT data plane operations which are checked against IoT Policies associated with the certificate identifying your Greengrass Core device. The TokenExchangeRole is not used for operations over the IoT data plane, meaning you don't need an IAM policy with permissions for shadow operations.

The previously linked document explains how to update an IoT Policy.

AWS IOT ShadowManager - Synchronization error

3 Answers3