1

How do I pipe the output of aws glue list-jobs... to aws glue start-job-run in powershell and bash?

Ex. something like:

aws glue list-jobs --tags name=something (magic here) | aws glue start-job-run (magic here)

This being a solution to the problem: quickly starting multiple jobs using the universal aws cli.

Thank you.

ams1
  • 113
  • 8

1 Answers1

1

  • Make the aws glue list-jobs call output the job names as text (--output text).

  • Using % (a built-in alias of the ForEach-Object cmdlet), pass each job name to aws glue start-job-run via the --job-name parameter and the automatic $_ variable.

# % is short for ForEach-Object
aws glue list-jobs --tags name=something --output text | 
  % { aws glue start-job-run --job-name $_ }

Update: You report that you ended up using the following, relying on the default output format, JSON:

# % is short for ForEach-Object
aws glue list-jobs --tags name=something  | 
  ConvertFrom-Json |
  Select-Object -ExpandProperty JobNames |
  % { aws glue start-job-run --job-name $_ }
mklement0
  • 382,024
  • 64
  • 607
  • 775
  • Thank you, eventually i ended up wth `aws glue list-jobs --tags name=something | ConvertFrom-Json | Select-Object -ExpandProperty JobNames | % {aws glue start-job-run --job-name $_}` – ams1 Nov 12 '22 at 19:42
  • Thanks, @ams1 - I can't verify it myself, but doesn't my command do the same? Or are the names not output line by line with `--output text`? – mklement0 Nov 12 '22 at 19:45