2

I am new to Glue job and followed the way to configure whl file as per the below link

Import failure of s3fs library in AWS Glue

I am getting the following error for the AWS Glue Python - 3 job

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fdac61a8080>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/fsspec/

ERROR: Could not find a version that satisfies the requirement fsspec==2021.07.0 (from s3fs==2021.7.0) (from versions: none)

ERROR: No matching distribution found for fsspec==2021.07.0 (from s3fs==2021.7.0)

Traceback (most recent call last):
  File "/tmp/runscript.py", line 207, in <module>
    download_and_install(args.extra_py_files, timeout=time_out)
  File "/tmp/runscript.py", line 141, in download_and_install
    install_package(file_extension, install_path, local_file_path, timeout=timeout)
  File "/tmp/runscript.py", line 115, in install_package
    install_wheel_with_retry(install_path, local_file_path, timeout=timeout)
  File "/tmp/runscript.py", line 108, in install_wheel_with_retry
    retry(run_check_call, install_path, local_file_path, timeout, exceptions=TimeoutExpired, message="{} installation".format(local_file_path))
  File "/tmp/runscript.py", line 35, in retry
    return func(*func_args)
  File "/tmp/runscript.py", line 106, in run_check_call
    check_call([sys.executable, "-m", "pip", "install", "--target={}".format(_install_path), _local_file_path], timeout=_timeout)
  File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-lw4p0gdn/s3fs-2021.7.0-py3-none-any.whl']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/tmp/runscript.py", line 230, in <module>
    raise e_type(e_value).with_traceback(new_stack)
TypeError: __init__() missing 1 required positional argument: 'cmd'
Phani
  • 803
  • 8
  • 11

1 Answers1

0

I think you just have a typo in your dependency that you are trying to install.

Compare:

2021.07.0
2021.7.0

So update the version of the s3fs you want to install.

See: https://pypi.org/project/s3fs/

Jens
  • 20,533
  • 11
  • 60
  • 86
  • I downloaded the latest file from here - https://pypi.org/project/s3fs/#files , and it has 2021.7.0. Are you saying to update file name from s3fs-2021.7.0-py3-none-any.whl to s3fs-2021.07.0-py3-none-any.whl – Phani Aug 05 '21 at 17:30