I would like to test a glue job with Moto ( https://docs.getmoto.org/en/latest/docs/services/glue.html ).
So I first start by creating the glue job:
@mock_glue
@mock_s3
class TestStringMethods(unittest.TestCase):
...
...
self.s3_client.upload_fileobj(open("etl.py", "rb"),self.s3_bucket_name, "etl.py")
self.glue_client = boto3.client('glue')
self.glue_client.create_job(
Name="Test Monitoring Job",
Role="test_role",
Command=dict(Name="glueetl", ScriptLocation=f"s3://{self.s3_bucket_name}/etl.py"),
GlueVersion='2.0',
NumberOfWorkers=1,
WorkerType='G.1X'
)
#Job is created with correct confs
assert (job["Name"] == job_name)
assert (job["GlueVersion"] == "2.0")
Then I proceed to launch it:
job_run_response = self.glue_client.start_job_run(
JobName=job_name,
Arguments={...} )
and get the job run:
response = self.glue_client.get_job_run(
JobName=job_name,
RunId=job_run_response['JobRunId']
)
However, at this point I find out that the configuration of the job I just launched is not the same of what I have defined in the create job. Look for example at the Glue version that I asserted before.
print(response)
{'JobRun': {'Id': '01',... 'Arguments': {'runSpark': 'spark -f test_file.py'}, 'ErrorMessage': '', 'PredecessorRuns': [{'JobName': 'string', 'RunId': 'string'}] ... 'GlueVersion': '0.9' }
There's no evidence my code has actually run, it looks like to be some standard default, apart from the job name. My questions basically are:
- have you experience with moto supporting this functionality?
- if yes, can you recognise if something's off with this code?