Getting current AWS Data Pipeline status from Java

Question

I am trying to access the current status of a data pipeline from Java Data Pipeline client. My use case is to activate a pipeline and wait till it's in completed state. I tried the answer from this thread: AWS Data Pipeline - Components, Instances and Attempts and Pipeline Status but I am only getting the current state as Scheduled even though the pipeline is in running state. This my code snippet:

DescribePipelinesRequest describePipelinesRequest = new DescribePipelinesRequest();
    describePipelinesRequest.setPipelineIds(Arrays.asList(pipelineId));
    final DescribePipelinesResult describePipelinesResult =
        dataPipelineClient.describePipelines(describePipelinesRequest);
    final List<Field> testPipeline =
        describePipelinesResult.getPipelineDescriptionList().get(0).getFields();
    for (Field field : testPipeline) {
      log.debug("Field: {} and {}", field.getKey(), field.getStringValue());
      if (field.getKey().equals("@pipelineState")) {
        log.debug("Pipeline state current: {} and {}", field.getStringValue());
      }
    }

Has anyone faced issues like this before? Btw, this pipeline has been made a on trigger pipeline scheduled to run every 100 years. We need to trigger this pipeline manually.

jmp · Answer 1 · 2020-01-11T21:43:30.533

I'm not sure this does exactly what you want but should help to point you in the right direction. You'll need the query the objects in the pipeline and get their status. These are what actually are running.

Java code

String pipelineid = "df-06036888777666777";//replace with your pipeline id
DataPipelineClient client = new DataPipelineClient();
QueryObjectsResult tasks = client.queryObjects( new QueryObjectsRequest().withPipelineId(pipelineid).withSphere("INSTANCE"));
DescribeObjectsResult results = client.describeObjects(new DescribeObjectsRequest().withObjectIds(tasks.getIds()).withPipelineId(pipelineid));

for (PipelineObject obj : results.getPipelineObjects()){
    for (Field field : obj.getFields()){
        if (field.getKey().equals("@status") && !field.getStringValue().equals("FINISHED") ){
            System.out.println(obj.getName() + " is still running...");
        }
    }
}

OUTPUT:

@CliActivity_2020-01-11T21:34:45 is still running...
@Ec2Instance_2020-01-11T21:34:45 is still running...

what you're doing currently is getting the pipeline information which will only show that it's been created successfully and scheduled.

We need to trigger this pipeline manually.

To do this activate the pipeline again. This will create new task objects which Data Pipeline will start to process. Currently as described this is an on-demand pipeline which will only create new tasks when it's activated manually.

Getting current AWS Data Pipeline status from Java

1 Answers1