0

I would like to test a glue job with Moto ( https://docs.getmoto.org/en/latest/docs/services/glue.html ).

So I first start by creating the glue job:

@mock_glue
@mock_s3
class TestStringMethods(unittest.TestCase):
...
...
self.s3_client.upload_fileobj(open("etl.py", "rb"),self.s3_bucket_name, "etl.py")
self.glue_client = boto3.client('glue')
self.glue_client.create_job(
            Name="Test Monitoring Job",
            Role="test_role",
            Command=dict(Name="glueetl", ScriptLocation=f"s3://{self.s3_bucket_name}/etl.py"),
            GlueVersion='2.0',
            NumberOfWorkers=1,
            WorkerType='G.1X'
        )
 #Job is created with correct confs
 assert (job["Name"] == job_name)
 assert (job["GlueVersion"] == "2.0")

Then I proceed to launch it:

job_run_response = self.glue_client.start_job_run(
        JobName=job_name,
        Arguments={...} ) 

and get the job run:

 response = self.glue_client.get_job_run(
        JobName=job_name,
        RunId=job_run_response['JobRunId']
    )

However, at this point I find out that the configuration of the job I just launched is not the same of what I have defined in the create job. Look for example at the Glue version that I asserted before.

print(response)
{'JobRun': {'Id': '01',... 'Arguments': {'runSpark': 'spark -f test_file.py'}, 'ErrorMessage': '', 'PredecessorRuns': [{'JobName': 'string', 'RunId': 'string'}] ...  'GlueVersion': '0.9' }

There's no evidence my code has actually run, it looks like to be some standard default, apart from the job name. My questions basically are:

  1. have you experience with moto supporting this functionality?
  2. if yes, can you recognise if something's off with this code?
LizardKing
  • 601
  • 6
  • 13
  • 1
    Moto only mocks the Glue service - it doesn't actually execute/run the code that you provide. The response for `get_job_run` is indeed mostly hardcoded: https://github.com/spulec/moto/blob/master/moto/glue/models.py#L1062 – Bert Blommers Nov 26 '22 at 11:00

0 Answers0