More that one object matches the predicate (2 in total) in AWS data pipeline

Question

In AWS data pipeline console, when I upload a pipeline definition file, I always get this error -

Pipeline creation failed. Data Pipeline failed to create pipeline : More that one object matches the predicate (2 in total). (Service: null; Status Code: 0; Error Code: null; Request ID: null)

Tried creating a pipeline definition by using 'Build from architect' option. It worked, then I exported the pipeline definition and tried creating another pipeline with 'Upload pipeline'. Did this just to be sure that the pipeline I created doesn't have any errors. But still got the same error. Any solutions?

score 0 · Answer 1 · answered Apr 09 '15 at 01:52

Pipeline console is always catching up when it comes to Data Pipeline Features when compared with the API as well as aws cli. Your best bet is to install and use aws cli to import pipeline definition.

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-cli-ref.html

We ran into a handful of scenarios where pipeline could not be imported through console -but would work fine when imported using aws cli.

score 0 · Accepted Answer · answered Apr 09 '15 at 10:26

The pipeline definition file expected by AWS varies in syntax when uploading pipeline definition as compared to pipeline definition one sees in 'Export' in Build in Architect.

The pipeline definition as explained in AWS documentation is of the format:

{
"objects" : [
    {
       "name1" : "value1",
       "name2" : "value2"
    },
    {
       "name1" : "value3",
       "name3" : "value4",
       "name4" : "value5"
    }
]}

This is the format that you can see when you try to Export pipeline definition after creating a pipeline using Build using Architect feature of Data pipeline.

But the format expected when you're Creating a pipeline by importing a pipeline definition file or using Boto is of the following format:

{u'parameterObjects': [], 
 u'pipelineObjects': [{u'fields': [{u'key': u'schedule',
                                u'refValue': u'ScheduleId_1'},
                               {u'key': u'scriptUri',
                                u'stringValue': u'S3://<Script path>'},
                               {u'key': u'workerGroup',
                                u'stringValue': u'EC2Cluster'},
                               {u'key': u'type',
                                u'stringValue': u'ShellCommandActivity'}],
                    u'id': u'ActivityId_1',
                    u'name': u'Activity 1'
                    }
                ]
}

More that one object matches the predicate (2 in total) in AWS data pipeline

2 Answers2