Python Step Functions API: get_activity_task seems to always timeout

Question

I've got a lambda function like this:

import boto3
import os
import json

step_functions = boto3.client('stepfunctions')
workers_topic = boto3.resource('sns').Topic(os.environ.get("WORKERS_TOPIC_ARN"))

def test_push_to_workers_sns(event, context):
    activity_response = \
        step_functions.get_activity_task(
            activityArn=os.environ.get("ACKNOWLEDGE_ACTIVITY_ARN"),
            workerName='test_push_to_workers_sns'
        )

    task_token, input_ = activity_response['taskToken'], activity_response['input']

    print(f"Task token is {task_token}")
    print(f"Input is {input}")

    if not task_token:
        print("No activity found")
        return

    workers_topic.publish(Message="blah blah")

When I set off an execution of the step function I have and it reaches the activity, I've repeatedly checked that running aws stepfunctions get-activity-task --activity-arn <ACKNOWLEDGE_ACTIVITY_ARN> on my terminal returns a taskToken and input that are both correct. However this lambda function seems to always time out regardless of whether or not the activity is running (I've got my timeout value set to 1 min 15 secs on the lambda function, and the activity state on the step function's timeout at 1 hour)

score 3 · Answer 1 · answered Mar 18 '18 at 12:00

I checked this case using following CloudFormation template and it works:

AWSTemplateFormatVersion: "2010-09-09"
Description: Stack creating AWS Step Functions state machine and lambda function calling GetActivityTask.

Resources:
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Handler: "index.handler"
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          import boto3
          import os
          import json

          step_functions = boto3.client('stepfunctions')
          workers_topic = boto3.resource('sns').Topic(os.environ.get("WORKERS_TOPIC_ARN"))


          def handler(event, context):
              activity_response = step_functions.get_activity_task(
                  activityArn=os.environ.get("ACKNOWLEDGE_ACTIVITY_ARN"),
                  workerName='test_push_to_workers_sns'
              )

              if 'taskToken' not in activity_response:
                  return

              task_token, task_input = activity_response['taskToken'], activity_response['input']

              print(f"Task token is {task_token}")
              print(f"Input is {input}")

              workers_topic.publish(Message="blah blah")
              step_functions.send_task_success(
                  taskToken=task_token,
                  output=task_input
              )

      Runtime: "python3.6"
      Timeout: 25
      Environment:
        Variables:
          WORKERS_TOPIC_ARN: !Ref WorkersTopic
          ACKNOWLEDGE_ACTIVITY_ARN: !Ref AcknowledgeActivity

  StateMachine:
    Type: AWS::StepFunctions::StateMachine
    Properties:
      RoleArn: !GetAtt StatesExecutionRole.Arn
      DefinitionString: !Sub
        - >
          {
            "Comment": "State Machine for GetActivityTask testing purposes.",
            "StartAt": "FirstState",
            "States": {
              "FirstState": {
                "Type": "Task",
                "Resource": "${ACKNOWLEDGE_ACTIVITY_ARN}",
                "End": true
              }
            }
          }
        - ACKNOWLEDGE_ACTIVITY_ARN: !Ref AcknowledgeActivity

  AcknowledgeActivity:
    Type: AWS::StepFunctions::Activity
    Properties:
      Name: !Sub ${AWS::AccountId}-AcknowledgeActivity

  WorkersTopic:
    Type: AWS::SNS::Topic

  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
          Action:
            - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: StepFunctionsAccess
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - states:GetActivityTask
                  - states:SendTaskFailure
                  - states:SendTaskSuccess
                Resource: arn:aws:states:*:*:*
        - PolicyName: SNSAccess
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - SNS:Publish
                Resource: arn:aws:sns:*:*:*

  StatesExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - !Sub states.${AWS::Region}.amazonaws.com
            Action: sts:AssumeRole
      Path: "/"
      Policies: []

I created execution manually from Step Functions console and executed Lambda manually from Lambda console.

Keep in mind that after you take taskToken using GetActivityTask the execution related with it is waiting for response (SendTaskSuccess or SendTaskFailure) until it reaches timeout (default is very long). So if you have taken token before then that execution isn't available for GetAcitivtyTask. You can lookup status of execution in Step Functions console looking at events for specific execution.

You should call SendTaskSuccess or SendTaskFailure from your code after getting token from GetActivityTask (otherwise execution will be hanging until it reaches timeout or is stopped).

Aside from original question: GetActivityTask is not designed to be called from Lambda. You can pass Lambda Function as resource to state machine (instead of activity) and it will be called when execution reaches specified state (event in handler will contain execution state). Activities should be used only for long-running jobs on dedicated machines (EC2, ECS). I should also point that there are service limits for GetActivityTask calls (25 RPS with bucket of size 1000) and Lambda-based states are limited basically only by transition count limit (400 per second with bucket of size 800). You can read more about step function limits here: https://docs.aws.amazon.com/step-functions/latest/dg/limits.html

Python Step Functions API: get_activity_task seems to always timeout

1 Answers1