4

My app has a S3 bucket with daily feeds, 2 DynamoDB tables that stores this data, an ELB application that exposes the JSON API to that data and a data pipeline flow that processes the incoming data and uploads into the tables.

My CloudFormation template currently builds the tables and the ELB app. I'd want the pipeline flow to be a part of the template too (so a complete staging environment, for example, will be built together).

I couldn't find references for that in the manuals. Does anyone know how to do it or can confirm that it's officially not supported?

Zach

Zach Moshe
  • 2,782
  • 4
  • 24
  • 40

1 Answers1

2

Editing my previous answer as Cloudformation has been supporting Data Pipeline.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-datapipeline-pipeline.html

As @SmartFingers noted, Data Pipeline is offered in fewer regions (although AWS has been adding) than Cloudformation itself.

https://aws.amazon.com/about-aws/whats-new/2014/02/20/aws-data-pipeline-now-available-in-four-new-regions/

user1452132
  • 1,758
  • 11
  • 21
  • that's what I was afraid of.. so if I need to create the pipeline based on output values from my template (dynamodb tables names for example), I'll have to create the template, get the values, amend the pipeline JSON file and activate it? – Zach Moshe Nov 25 '14 at 10:19
  • Yes. That is the only approach I know of. – user1452132 Nov 25 '14 at 21:45
  • 14
    CloudFormation supports Data Pipeline since June 11, 2015: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html – Hugo Lopes Tavares Jul 30 '15 at 22:45
  • 1
    Another issue is, Data pipeline is only supported in x-large regions. CFN doesn't support resource creation from a different region. – SmartFingers Oct 29 '20 at 02:48
  • Can nested stacks be used as defined here ? I did not get a chance. curious. https://aws.amazon.com/blogs/infrastructure-and-automation/multiple-account-multiple-region-aws-cloudformation/ – user1452132 Oct 29 '20 at 13:35