1

I am trying to create a Python Wheel for Great_Expectations. The .whl provided by Great_Expectations exists here https://pypi.org/project/great-expectations/#files - great-expectations 0.13.25. Unfortunately, it appears that this .whl doesn't contain all the libraries I need to in order to work with Great_Expectations in Azure Synapse Apache Spark Pool.

Therefore, it looks like I will either have to create my own Great_Expectations package a python project with all of its dependencies for offline install.whl or at the very least try and establish what libraries are contained within the existing package great-expectations 0.13.25

Therefore, can someone let me know how to create a Python Wheel(ie. Python Package, with all of its dependencies for Great_Expectations). Alternatively, can someone let me know how to determine what module/dependencies are contained with a package?

Thanks

Patterson
  • 1,927
  • 1
  • 19
  • 56
  • A `.whl` is simply a `.zip` file of the package with the extension renamed. – Diptangsu Goswami Aug 04 '21 at 09:55
  • 1
    " it appears that this .whl doesn't contain all the libraries I need" – are you aware that a wheel is *just* the package, not its dependencies? – MisterMiyagi Aug 04 '21 at 09:55
  • If there are extra dependencies, you need to download each `.whl` using `pip download ` and then install from the downloaded wheels. – Diptangsu Goswami Aug 04 '21 at 09:56
  • @DiptangsuGoswami, I think understand your point about having to download each .whl, but its a little unclear about 'install from the downloaded wheels'? – Patterson Aug 04 '21 at 10:05

2 Answers2

1

To add new dependencies, update requirements.txt (You actually need to update install_requires in setup.py, but in this project they are reading the requirements file to fetch the requirements)

You will need to clone the git repo so as to update that list.

Then to create a new wheel out of that source, just run:

python setup.py bdist_wheel

(You may need to run pip install wheel if wheel doesn't exist)

Docs: wheel

To the second question: What modules / libraries are within a Python Wheel?

Just the package, the dependencies are installed from sources when you install the package.

Ani Menon
  • 27,209
  • 16
  • 105
  • 126
  • Hi Ani, this is amazing. Thank you. Just on the point of libraries/dependencies already included. Will the **great_expectations-0.13.25-py3-none-any.whl** already have the required libraries to at least get you up and running with Great_Expectations in Azure Synapse Apache Spark? – Patterson Aug 04 '21 at 12:41
  • Should I be able to run ```from setuptools import find_packages, setup``` directly from Apache Spark? Or will I need to install the setup.py as a workspace package in Azure Synapse Apache Spark Pool? I hope I'm making sense – Patterson Aug 04 '21 at 12:44
  • Dependencies aren't packaged with your package(whl) even here, all that we are doing is asking python to install the requirements by downloading from appropriate source while installing the package(whl). – Ani Menon Aug 04 '21 at 18:14
  • Create the whl with given command then run the pip command on your remote to install the `whl` there.. pip will take care of installing required dependencies as far as your remote can download from PyPI (or some index). – Ani Menon Aug 04 '21 at 18:18
-1

Consider to use Conda pack. It was explicitly created for such a use case of making Python / Conda packages easily portable

Georg Heiler
  • 16,916
  • 36
  • 162
  • 292