Boto3 DMS 'modify_replication_task' error on replication task settings JSON

Question

I'm using boto3 to create DMS replication tasks. I'm using the following replication_task_settings.json to create the replication tasks:

    {
      "TargetMetadata": {
          "TargetSchema": "",
          "SupportLobs": true,
          "FullLobMode": false,
          "LobChunkSize": 0,
          "LimitedSizeLobMode": true,
          "LobMaxSize": 256,
          "InlineLobMaxSize": 0,
          "LoadMaxFileSize": 0,
          "ParallelLoadThreads": 0,
          "ParallelLoadBufferSize": 0,
          "BatchApplyEnabled": false,
          "TaskRecoveryTableEnabled": false
      },
      "FullLoadSettings": {
          "TargetTablePrepMode": "TRUNCATE_BEFORE_LOAD",
          "CreatePkAfterFullLoad": false,
          "StopTaskCachedChangesApplied": false,
          "StopTaskCachedChangesNotApplied": false,
          "MaxFullLoadSubTasks": 8,
          "TransactionConsistencyTimeout": 1000,
          "CommitRate": 10000
      },
      "Logging": {
          "EnableLogging": true,
          "LogComponents": [
              {
                  "Id": "SOURCE_UNLOAD",
                  "Severity": "LOGGER_SEVERITY_DEFAULT"
              },
              {
                  "Id": "TARGET_LOAD",
                  "Severity": "LOGGER_SEVERITY_DEFAULT"
              },
              {
                  "Id": "SOURCE_CAPTURE",
                  "Severity": "LOGGER_SEVERITY_DEFAULT"
              },
              {
                  "Id": "TARGET_APPLY",
                  "Severity": "LOGGER_SEVERITY_DEFAULT"
              },
              {
                  "Id": "TASK_MANAGER",
                  "Severity": "LOGGER_SEVERITY_DEFAULT"
              }
          ],
      },
      "ControlTablesSettings": {
          "ControlSchema": "control",
          "HistoryTimeslotInMinutes": 5,
          "HistoryTableEnabled": true,
          "SuspendedTablesTableEnabled": true,
          "StatusTableEnabled": true
      },
      "StreamBufferSettings": {
          "StreamBufferCount": 3,
          "StreamBufferSizeInMB": 8,
          "CtrlStreamBufferSizeInMB": 5
      },
      "ChangeProcessingDdlHandlingPolicy": {
          "HandleSourceTableDropped": false,
          "HandleSourceTableTruncated": true,
          "HandleSourceTableAltered": false
      },
      "ErrorBehavior": {
          "DataErrorPolicy": "LOG_ERROR",
          "DataTruncationErrorPolicy": "LOG_ERROR",
          "DataErrorEscalationPolicy": "SUSPEND_TABLE",
          "DataErrorEscalationCount": 0,
          "TableErrorPolicy": "SUSPEND_TABLE",
          "TableErrorEscalationPolicy": "STOP_TASK",
          "TableErrorEscalationCount": 0,
          "RecoverableErrorCount": -1,
          "RecoverableErrorInterval": 5,
          "RecoverableErrorThrottling": true,
          "RecoverableErrorThrottlingMax": 1800,
          "ApplyErrorDeletePolicy": "IGNORE_RECORD",
          "ApplyErrorInsertPolicy": "LOG_ERROR",
          "ApplyErrorUpdatePolicy": "LOG_ERROR",
          "ApplyErrorEscalationPolicy": "LOG_ERROR",
          "ApplyErrorEscalationCount": 0,
          "ApplyErrorFailOnTruncationDdl": false,
          "FullLoadIgnoreConflicts": true,
          "FailOnTransactionConsistencyBreached": false,
          "FailOnNoTablesCaptured": false
      },
      "ChangeProcessingTuning": {
          "BatchApplyPreserveTransaction": true,
          "BatchApplyTimeoutMin": 1,
          "BatchApplyTimeoutMax": 30,
          "BatchApplyMemoryLimit": 500,
          "BatchSplitSize": 0,
          "MinTransactionSize": 1000,
          "CommitTimeout": 1,
          "MemoryLimitTotal": 1024,
          "MemoryKeepTime": 60,
          "StatementCacheSize": 50
      },
      "ValidationSettings": {
          "EnableValidation": true,
          "ValidationMode": "ROW_LEVEL",
          "ThreadCount": 5,
          "PartitionSize": 10000,
          "FailureMaxCount": 10000,
          "RecordFailureDelayInMinutes": 5,
          "RecordSuspendDelayInMinutes": 30,
          "MaxKeyColumnSize": 8096,
          "TableFailureMaxCount": 1000,
          "ValidationOnly": false,
          "HandleCollationDiff": false,
          "RecordFailureDelayLimitInMinutes": 0
      }
    }

The JSON above works fine when calling dms_client.create_replication_task. However, it doesn't work when modifying the replication tasks.

When calling dms_client.modify_replication_task with the replication_task_settings.json mentioned above I get the following error:

botocore.exceptions.ClientError: An error occurred (InvalidParameterValueException) when calling the ModifyReplicationTask operation: Invalid task settings JSON

I'm not sure why this is happening and any help would be greatly appreciated!

I tried removing some settings that are already defaulted. I tried looking for malformed JSON but nothing obvious.

I would expect the replication_task_settings,json to work for both creating and modifying DMS replication tasks.

score 1 · Answer 1 · edited Aug 30 '19 at 02:41

I had similar issue while trying to modify the replication task using CLI :

aws --profile non-prod dms modify-replication-task --replication-task-arn arn:aws:dms:ap-southeast-3:567384657322:task:ABC --replication-task-settings file:/json_task --region ap-southeast-3

This is the error :

An error occurred (InvalidParameterValueException) when calling the ModifyReplicationTask operation: Invalid task settings JSON

Below command worked :

aws --profile non-prod dms modify-replication-task --replication-task-arn arn:aws:dms:ap-southeast-3:567384657322:task:ABC --replication-task-settings file://json_task --region ap-southeast-3

Change is to use two "//" before JSON file name to modify the task.

score 1 · Answer 2 · answered Sep 13 '21 at 05:27

The boto3 document for modify_replication_task() for TableMappings suggests -- When using the CLI or boto3, provide the path of the JSON file that contains the table mappings. Precede the path with file:// . For example, --table-mappings file://mappingfile.json . When working with the DMS API, provide the JSON as the parameter value.

Yes, you are absolutely correct. Providing JSON file as a parameter works in CLI but not with boto3. In boto3, it throws the error 'Invalid TableMappings / Invalid JSON error'.

Try the below to make it work:


    import boto3
    dms_client = boto3.client('dms')
    table_mapping = {
    "rules":[
    {
    "rule-type":"selection",
    "rule-id":str(idx),
    "rule-name":str(idx),
    "object-locator":{
     "schema-name":schema_name,
     "table-name":table_name
     },
     "rule-action":"include"
     }
     ]
     }

dms_client.modify_replication_task(
ReplicationTaskArn=repl_task_arn,
TableMappings=json.dumps(table_mapping)
)

Hey, My json is in s3, how should I pass that json to table mapping parameter? — Tula, May 16 '22 at 07:57

score 0 · Answer 3 · answered Oct 20 '22 at 09:33

0

I faced a similar issue yesterday. My json is in S3 and the final solution was simply applying indent=4

    obj = s3.Object(bucket, key)
    new_json = obj.get()['Body'].read().decode('utf-8')

    new_json = json.loads(new_json)

    client.modify_replication_task(
    ReplicationTaskArn=taskarn,
    TableMappings=json.dumps(new_json, indent=4)
    )

answered Oct 20 '22 at 09:33

Jorge Sánchez Serna

1

Better indent the code snippet (lines between parenthesis) – 70ny Oct 23 '22 at 17:31

Boto3 DMS 'modify_replication_task' error on replication task settings JSON

3 Answers3