0

Does anyone have a working example of using firehose.update_destination to set an S3 destination's DataFormatConversionConfiguration? I'm following the guidance in Is it possible to specify data format conversion in AWS Cloudformation?, using boto3 (the AWS Python SDK), but I've not been successful. When I include a DFCC in an ExtendedS3DestinationConfiguration argument, it fails with the following error:

Exception during processing: An error occurred (InvalidArgumentException) when calling the UpdateDestination operation: RoleArn must not be null or empty

If I pass the original destination configuration (as returned by describe_delivery_stream) unchanged, the update succeeds. I can also change other config options, e.g. BufferingHints. The only time it fails is when DataFormatConversionConfiguration is non-null.

For example, passing this works:

{
  "RoleARN": "arn:aws:iam::1234567:role/MyExecutionRole",
  "BucketARN": "arn:aws:s3:::my-bucket",
  "Prefix": "databases/tables/requests/",
  "BufferingHints": {
    "SizeInMBs": 64,
    "IntervalInSeconds": 120
  },
  "CompressionFormat": "UNCOMPRESSED",
  "EncryptionConfiguration": {
    "NoEncryptionConfig": "NoEncryption"
  },
  "CloudWatchLoggingOptions": {
    "Enabled": false
  },
  "S3BackupMode": "Disabled"
}

but passing this fails:

{
  "RoleARN": "arn:aws:iam::1234567:role/MyExecutionRole",
  "BucketARN": "arn:aws:s3:::my-bucket",
  "Prefix": "databases/tables/requests/",
  "BufferingHints": {
    "SizeInMBs": 64,
    "IntervalInSeconds": 120
  },
  "CompressionFormat": "UNCOMPRESSED",
  "EncryptionConfiguration": {
    "NoEncryptionConfig": "NoEncryption"
  },
  "CloudWatchLoggingOptions": {
    "Enabled": false
  },
  "S3BackupMode": "Disabled",
  "DataFormatConversionConfiguration": {
    "InputFormatConfiguration": {
      "Deserializer": {
        "OpenXJsonSerDe": {
        }
      }
    },
    "SchemaConfiguration": {
      "TableName": "requests",
      "DatabaseName": "mydb"
    },
    "OutputFormatConfiguration": {
      "Serializer": {
        "OrcSerDe": {
        }
      }
    }
  }
}

The only difference is the DataFormatConversionConfiguration element.

Am I overlooking something obvious? Perhaps the DFCC element is malformed? I've not been able to find any working examples, so I'm going purely from documentation.

I'm also rather surprised by the use of RoleARN and BucketARN in the input element, vs the usual convention of RoleArn and BucketArn, but not sure if it's germane.

Peter Halverson
  • 380
  • 2
  • 12

1 Answers1

1

As you suspected, your DataFormatConversionConfiguration is malformed.

Perhaps confusingly I think the RoleArn it's complaining about being missing is DataFormatConversionConfiguration.SchemaConfiguration.RoleARN.

I'm not going to copy it all here, but I find looking at the service documentation is the best way to find deeper information about the types used by the SDK: https://docs.aws.amazon.com/firehose/latest/APIReference/API_DataFormatConversionConfiguration.html

thomasmichaelwallace
  • 7,926
  • 1
  • 27
  • 33
  • Thanks! And, yes, I've often figured out some obscure CFm error by comparing the SDK documentation, the base API reference, and the CLI documentation and triangulating; hadn't done that in this case. Of course, looking at API docs gets you only so far, since – inexplicably, maddeningly – CFn doesn't always support all the properties defined for the API. – Peter Halverson Nov 01 '18 at 14:58
  • Yeah, CFn often lags behind (a fact not helped by AWS' spotty docs!). I guess because it's maintained by a different team- which seems like a weird choice to me- but who am I to argue with AWS' success! Good luck! – thomasmichaelwallace Nov 01 '18 at 15:01