How to use jsonPath inside array in AWS Step Functions

Question

I am writing an AWS step function, and for one of the steps, I wish to call a lambda that accepts an array as one of the inputs. However, if I try to pass in a JsonPath into the array, I get

The value for the field 'arrayField.$' must be a STRING that contains a JSONPath but was an ARRAY

My step function definition:

{
  "StartAt": "First",
  "States": {
  "First": {
    "Type": "Pass",
    "Parameters": {
      "type": "person"
    },
    "ResultPath": "$.output",
    "Next": "Second"
  },
    "Second": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
      "Parameters": {
        "regularParameter": "some string",
        "arrayParameter.$": ["$.output.type"]
      },
      "Next": "Succeed"
    },
    "Succeed": {
      "Type": "Succeed"
    }
  }
}

How can I use jsonPath inside the array?

score 24 · Accepted Answer · answered Sep 02 '20 at 20:21

24

Since a new release you could use the intrinsic function States.Array:

  "arrayParameter.$": "States.Array($.output.type)"

https://docs.aws.amazon.com/step-functions/latest/dg/amazon-states-language-intrinsic-functions.html

answered Sep 02 '20 at 20:21

Petar Butkovic

1,820
15
14

5

It solved my problem when needing one static and one dynamic array element "Args.$": "States.Array('fixed_text',$.ProcessName)" – andresv Sep 09 '20 at 14:52
Hi, I want to pass a list of variables "args": ["$.variable1", "$.variable2", "$.variable3"] but it's giving me an error – Abdul Haseeb Dec 15 '20 at 16:38
@AbdulHaseeb Have you tried "args": "States.Array($.variable1, $.variable2, $.variable3)"? – Petar Butkovic Jan 05 '21 at 14:17
This also works in CDK `step_fn.JsonPath.string_at('States.Array($.output.type)')` – ollik1 Oct 06 '21 at 06:39

score 1 · Answer 2 · answered Apr 10 '20 at 13:53

As @Seth Miller mentioned JsonPath resolution within arrays doesn't work unfortunately. If the amount of values to replace in the array is small and known there's a simple workaround (in my case I needed an array of size 1).

The steps are:

Initialise the array with the number of values you need;
Replace each value using "ResultPath": "$.path.to.array[n]";
Use "$.path.to.array" in your task.

Simple, working example:

{
  "StartAt": "First",
  "States": {
    "First": {
      "Type": "Pass",
      "Parameters": {
        "type": "person"
      },
      "ResultPath": "$.output",
      "Next": "Initialise Array"
    },
    "Initialise Array": {
      "Comment": "Add an entry for each value you intend to have in the final array, the values here don't matter.",
      "Type": "Pass",
      "Parameters": [
        0
      ],
      "ResultPath": "$.arrayParameter",
      "Next": "Fill Array"
    },
    "Fill Array": {
      "Comment": "Replace the first entry of array with parameter",
      "Type": "Pass",
      "InputPath": "$.output.type",
      "ResultPath": "$.arrayParameter[0]",
      "End": true
    }
  }
}

And to use the resulting array in your task example:

    "Second": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
      "Parameters": {
        "regularParameter": "some string",
        "arrayParameter.$": "$.arrayParameter"
      },
      "Next": "Succeed"
    },

Dmitry Kolomiets · Answer 3 · 2020-06-18T13:41:57.583

Another way to approach this is by using Parallel state that outputs an array of objects and then use jsonPath to convert it to a simple array:

{
  "StartAt": "Parallel",
  "States": {
    "Parallel": {
      "Type": "Parallel",
      "Next": "Use Array",
      "ResultPath": "$.items",
      "Branches": [
        {
          "StartAt": "CreateArray",
          "States": {
            "CreateArray": {
              "Type": "Pass",
              "Parameters": {
                "value": "your value"
              },
              "End": true
            }
          }
        }
      ]
    },
    "Use Array": {
      "Type": "Pass",
      "Parameters": {
        "items.$": "$.items[*].value"
      },
      "End": true
    }
  }
}

In this example, Parallel state outputs the following json:

{
  "items": [
    {
      "value": "your value"
    }
  ]
}

And "Use Array" state produces:

{
  "items": [
    "your value"
  ]
}

score 0 · Answer 4 · answered Dec 04 '19 at 12:40

0

JSONPath inside parameters field need to be a string. So if you want to pass to lambda function a parameter called arrayParameter, you´ll need to make a jsonPath query that extract that array.

For example, if inside the key output is a key called outputArray with the array as its value.

Input JSON:

{
  "pre": "sdigf",
  "output": {
    "result": 1,
    "outputArray": ["test1","test2","test.."]
  }
}

The parameter sintax:

"arrayParameter.$": "$.output.outputArray"

answered Dec 04 '19 at 12:40

WaterKnight

197
7

1

Right, but this way, the contents of the array are constants; I'm looking for a way to include jsonPaths inside the array. – alexgbelov Dec 04 '19 at 18:18
The output array could be generated dynamically in the first state if it was of Type Task. All you need to do is create a Lambda function for the first state that generates the array dinamically. – WaterKnight Dec 05 '19 at 09:32

score 0 · Answer 5 · answered Apr 09 '20 at 04:17

Reasonable advice

I ran into a use case for JsonPath resolution within arrays today and found (like you have) that the functionality does not exist today. I ended up deciding that doing the data massaging in code was simpler and cleaner. For example, you could create a small Lambda that takes in the object emitted by First and massages it to a format acceptable to Second and adds it to the output (WaterKnight mentions this solution in a comment to another question).

This assumes that you are, for some reason, unable to change the format of the input to that Lambda in Second (which would be the absolute shortest path here).

Unreasonable advice

That said, if you want a way to do this completely within Step Functions that is fairly gross, you can use the result of a Map state that executes Pass states. The output of the Map state is an array that aggregate the output of each constituent Pass state. These Pass states simply emit the value(s) you want in the final array using the Parameters attribute. An example Step Function definition follows. I did warn that it is gross and that I went a different way to solve the problem.

 {
    "StartAt": "First",
    "Comment": "Please don't actually do this",
    "States": {
      "First": {
        "Type": "Pass",
        "Parameters": {
          "type": "person"
        },
        "ResultPath": "$.output",
        "Next": "Add Array"
      },
      "Add Array": {
        "Comment": "A Map state needs some array to loop over in order to work. We will give it a dummy array. Add an entry for each value you intend to have in the final array. The values here don't matter.",
        "Type": "Pass",
        "Result": [
          0
        ],
        "ResultPath": "$.dummy",
        "Next": "Mapper"
      },
      "Mapper": {
        "Comment": "Add a Pass state with the appropriate Parameters for each field you want to map into the output array",
        "Type": "Map",
        "InputPath": "$",
        "ItemsPath": "$.dummy",
        "Parameters": {
          "output.$": "$.output"
        },
        "Iterator": {
          "StartAt": "Massage",
          "States": {
            "Massage": {
              "Type": "Pass",
              "Parameters": {
                "type.$": "$.output.type"
              },
              "OutputPath": "$.type",
              "End": true
            }
          }
        },
        "ResultPath": "$.output.typeArray",
        "Next": "Second"
      },
      "Second": {
        "Comment": "The Lambda in your example is replaced with Pass so that I could test this",
        "Type": "Pass",
        "Parameters": {
          "regularParameter": "some string",
          "arrayParameter.$": "$.output.typeArray"
        },
        "Next": "Succeed"
      },
      "Succeed": {
        "Type": "Succeed"
      }
    }
  }

score 0 · Answer 6 · answered May 05 '20 at 18:56

As many answers correctly pointed out, it's not possible to do it exactly the way you need. But I would suggest another solution: an array of dictionaries. It's not exactly what you need, but is native and not hacky.

"Second": {
  "Type": "Task",
  "Resource": "arn:aws:lambda:us-east-1:<aws_id>:function:MyFunction",
  "Parameters": {
    "regularParameter": "some string",
    "arrayParameter": [{"type.$": "$.output.type"}]
  },
  "Next": "Succeed"
},

The result would be

{
  "regularParameter": "some string",
  "arrayParameter": [{"type": "SingleItemWrappedToAnArray"}]
}

Realized that I provided an answer that is very similar to yours. Once you have an array of objects, you can use jsonPath magic to convert it into array of values, in your case it would be: $.arrayParameter[*].type — Dmitry Kolomiets, Jun 18 '20 at 11:55

How to use jsonPath inside array in AWS Step Functions

6 Answers6

Reasonable advice

Unreasonable advice

Linked