1

I am trying to set up the AWS Chatbot with Slack integrations to display error messages for changes in states (errors) for AWS Glue. I have set up AWS EventBridge event pattern to catch Glue Job State Changes as follows:

{
  "source": ["aws.glue"],
  "detail-type": ["Glue Job State Change"],
  "detail": {
    "state": [
      "FAILED"
    ]
  }
}

This successfully catches all failed Glue Jobs and I have set up an AWS SNS topic as the target using the input transformer.

Input Transformer Input Path

{"jobname":"$.detail.jobName","jobrunid":"$.detail.jobRunId","jobstate":"$.detail.state"}

Input Transformer Input Template

"{\"detail-type\": \"Glue Job <job-name> has entered the state <job-state> with the message <message>.\"}"

AWS SNS has a subscriptions endpoint to the AWS Chatbot which fails to send the notification to Slack.

AWS Chatbot CloudWatch logs after an event using Input Transformer

Event received is not supported (see https://docs.aws.amazon.com/chatbot/latest/adminguide/related-services.html ): 
{
    "subscribeUrl": null,
    "type": "Notification",
    "signatureVersion": "1",
    "signature": <signature>,
    "topicArn": <topic-arn>,
    "signingCertUrl": <signing-cert-url>,
    "messageId": <message-id>,
    "message": "{\"detail-type\": \"Glue Job MyJob has entered the state FAILED with the message SystemExit: None.\"}",
    "subject": null,
    "unsubscribeUrl": <unsubscribe-url>,
    "timestamp": "2022-03-02T12:17:16.879Z",
    "token": null
}

When the input is set to 'Matched Events' in the AWS EventBridge Select Target, the Slack Notification will send however it lacks any details.

Slack Notification

Glue Job State Change | eu-west-1 | Account: <account>
Glue Job State Change

AWS EventBridge Matched Events JSON Output

{
    "Type" : "Notification",
    "MessageId" : <message-id>,
    "TopicArn" : <topic-arn>,
    "Message" : "{\"detail-type\": [\"Glue Job State Change\"]}",
    "Timestamp" : "2022-03-02T11:17:52.443Z",
    "SignatureVersion" : "1",
    "Signature" : <signature>,
    "SigningCertURL" : <signing-cert-url>,
    "UnsubscribeURL" : <unsubscribe-url>
  }

There are very little differences between the two JSON outputs however the input transformer is considered an unsupported event. Is it possible to generate a custom message when using the AWS Chatbot for errors?

samuel
  • 23
  • 5
  • Looks like there are limitations of the AWS Chatbot listed in another post: https://stackoverflow.com/questions/65854288/can-i-customize-messages-received-through-slacks-aws-chatbot-integration – samuel Mar 02 '22 at 13:50

1 Answers1

0

The best solution was to create a Lambda function as the target of the AWS EventBridge which performs a POST to a Slack Webhook.

# Import modules
import logging
import json
import urllib3

# Set up logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)


# Define Lambda function
def lambda_handler(event, context):
    
    http = urllib3.PoolManager()
    
    url = <url>
    link = <glue-studio-monitoring-link>
    message = f"A Glue Job {event['detail']['jobName']} with Job Run ID {event['detail']['jobRunId']} has entered the state {event['detail']['state']} with error message: {event['detail']['message']}. Visit the link for job monitoring {link}"
    logger.info(message)
    headers = {"Content-type": "application/json"}
    data = {'text': message}
    response = http.request('POST',
                        url,
                        body = json.dumps(data),
                        headers = headers,
                        retries = False)
    logger.info(response.status)
samuel
  • 23
  • 5