I want to use Python Fabric with my custom Operator, how should I install fabric on workers?

Question

At this point I'm thinking about calling bash command pip install fabric2 each time my operator executed, but this does not looke like a good idea.

Answer 1

Create a requirements.txt file similar and pass that as a variable while creating the cloud composer enviroment.

Sample requirements.txt file:

scipy>=0.13.3
scikit-learn
nltk[machine_learning]

Pass the requirements.txt file to the environments.set-python-dependencies command to set your installation dependencies.

gcloud beta composer environments update ENVIRONMENT_NAME \
--update-pypi-packages-from-file requirements.txt \
--location LOCATION

Answer 2

Turns out you can use: PythonVirtualenvOperator it supports pip deps.

Another option that is available for the Composer users is to install deps via the composer itself: https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies