0

How to choose python version 3 while deploying AWS glue Job with glue version 1.0 using YAML(serverless)? I'm deploying AWS glue using serverless YAML code. AWS has provided GlueVersion parameter to choose the version of glue to use which I'm setting to '1.0'. I want to use Python 3 but it is deploying to python 2 in glue job. Does any have an idea how to set it to use python3 using YAML deployment code without manual changes in the glue job?

Rajan Beri
  • 1
  • 4
  • there is a property pythonversion as shown in https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/glue.html#Glue.Client.create_job. you should also find equivalent – Prabhakar Reddy Nov 24 '20 at 05:07
  • the python version property mentioned in docs is for glue python shell job, not glue spark job – Rajan Beri Nov 24 '20 at 14:09

1 Answers1

0

Confirmed with AWS, their document is confusing where they mentioned PythonVersion parameter is for python shell. They are working on correcting the wording of the document. Below could formation template work for me.

Type: AWS::Glue::Job
Properties:
  Name: <job name>
  Description: ETL Job
  GlueVersion: '1.0'
  Command:
    Name: glueetl
    PythonVersion: 3
    ScriptLocation: <script location>
  ExecutionProperty:
    MaxConcurrentRuns: 1
  AllocatedCapacity: <number for workers>
  DefaultArguments:
   <any default arguments for glue job>
  Role:
    Ref: <glue roles>
  Connections:
    Connections:
      - Ref: <any JDBC connections>
Rajan Beri
  • 1
  • 4