AWS CodePipeline Custom Lambda Function Runs Forever and Never Returns

Question

I have a simple AWS CodePipeline with the standard "Source" -> "Build" -> "Deploy" pipeline stages that work fine and I am trying to add my own custom final pipeline stage that is a single AWS Lambda Function. The problem is my last, custom Lambda function runs multiple times and after a very long time, errors with the following message:

Please see the attached screenshot for the whole pipeline:

When the pipeline reaches this final step, it spins for a very long time with the "Blue ( In-Progress )" status before showing an error as shown here:

Here is my Lambda Function code:

from __future__ import print_function
import hashlib
import time
import os
import boto3
import json
from botocore.exceptions import ClientError

def lambda_handler(event, context):
    
    # Test
    AWS_ACCESS_KEY = ASDF1234
    AWS_SECRET_KEY = ASDF1234
    SQS_TESTING_OUTPUT_STATUS_QUEUE_NAME = 'TestingOutputQueue'

    # Get the code pipeline
    code_pipeline = boto3.client('codepipeline')
    
    # Get the job_id
    for key, value in event.items():
        print(key,value)
    job_id = event['CodePipeline.job']['id']
    DATA = json.dumps(event)
    
    # Create a connection the SQS Notification service
    sqs_resource_connection = boto3.resource(
        'sqs',
        aws_access_key_id = AWS_ACCESS_KEY,
        aws_secret_access_key = AWS_SECRET_KEY,
        region_name = 'us-west-2'
    )
    
    # Get the queue handle
    print("Waiting for notification from AWS ...")
    queue = sqs_resource_connection.get_queue_by_name(QueueName = SQS_TESTING_OUTPUT_STATUS_QUEUE_NAME)
    messageContent = ""
    cnt = 1
        
    # Replace sender@example.com with your "From" address.
    # This address must be verified with Amazon SES.
    SENDER = ME
    
    # Replace recipient@example.com with a "To" address. If your account
    # is still in the sandbox, this address must be verified.
    RECIPIENTS = [YOU]
    
    # If necessary, replace us-west-2 with the AWS Region you're using for Amazon SES.
    AWS_REGION = "us-east-1"
    
    # The subject line for the email.
    SUBJECT = "Test Case Results"
    
    # The email body for recipients with non-HTML email clients.
    BODY_TEXT = ("Test Case Results Were ...")
    
    # The HTML body of the email.
    BODY_HTML = """<html>
    <head></head>
    <body>
      <h1>Amazon SES Test (SDK for Python)</h1>
      <p>%s</p>
    </body>
    </html>
                """%(DATA)
    
    # The character encoding for the email.
    CHARSET = "UTF-8"
    
    # Create a new SES resource and specify a region.
    client = boto3.client('ses', region_name=AWS_REGION)
    
    # Try to send the email.
    try:
        # Provide the contents of the email.
        response = client.send_email(
            Destination={
                'ToAddresses': RECIPIENTS,
            },
            Message={
                'Body': {
                    'Html': {
                        'Charset': CHARSET,
                        'Data': BODY_HTML,
                    },
                    'Text': {
                        'Charset': CHARSET,
                        'Data': BODY_TEXT,
                    },
                },
                'Subject': {
                    'Charset': CHARSET,
                    'Data': SUBJECT,
                },
            },
            Source=SENDER,
            # If you are not using a configuration set, comment or delete the
            # following line
            #ConfigurationSetName=CONFIGURATION_SET,
        )
    # Display an error if something goes wrong.
    except ClientError as e:
        code_pipeline.put_third_party_job_failure_result(jobId=job_id, failureDetails={'message': message, 'type': 'JobFailed'})
        code_pipeline.put_job_failure_result(jobId=job_id, failureDetails={'message': message, 'type': 'JobFailed'})      
        print(e.response['Error']['Message'])
    else:
        code_pipeline.put_third_party_job_success_result(jobId=job_id)
        code_pipeline.put_job_success_result(jobId=job_id)
        print("Email sent! Message ID:"),
        print(response['MessageId'])
        
    print('Function complete.')   
    return "Complete."

How can I get the Lambda to fire once and return so the pipeline can complete properly.

For some reason I didn't see your code calling `put_job_success_result` the 1st time I read your Q. Anyway, you don't need to call `put_third_party_job_success_result`. Try without it and add some debug prints to check if it hangs and where. — jweyrich, Mar 27 '19 at 21:30
Hi @jweyrich - thanks for the answer and comment. I have tried without the "third_party_job" calls, but it still fails. I must be missing something else but I cannot yet figure it out. I have tried various combinations of "put_third_party_job_success_result", "put_job_success_result", and even having a "return". Any other ideas? Thanks. — PhilBot, Mar 28 '19 at 11:40
When it run indefinitely, it sends multiple emails - so it acts as though the lambda function is being called several times. Is that a clue? — PhilBot, Mar 28 '19 at 11:41
Is it possible for `put_job_success_result` to raise an exception? Move it inside the `try` block to validate. — jweyrich, Mar 28 '19 at 14:18

jweyrich · Accepted Answer · 2019-03-27T20:34:03.027

You are missing an important integration between your Lambda Function and the CodePipeline service.

You MUST notify CodePipeline about the result of your custom step, whether it succeeded or not - see my examples below.

Reporting success:

function reportSuccess(job_id) {
  var codepipeline = new AWS.CodePipeline();
  var params = {
    jobId: job_id,
  };
  return codepipeline.putJobSuccessResult(params).promise();
}

Reporting failure:

function reportFailure(job_id, invoke_id, message) {
  var codepipeline = new AWS.CodePipeline();
  var params = {
    failureDetails: {
      message: message,
      type: 'JobFailed',
      externalExecutionId: invoke_id,
    },
    jobId: job_id,
  };
  return codepipeline.putJobFailureResult(params).promise();
}

The integration was designed this way because one may want to integrate with an external job worker, in which their Lambda starts that worker (example, an approval process), and that worker then takes control and decides whether the whole step succeeded or failed.

AWS CodePipeline Custom Lambda Function Runs Forever and Never Returns

1 Answers1