3

I'm trying to install a private python package in the Google Cloud composer environment. I usually install the package using a personal access token. That works with usual pip.

pip install git+https://$TOKEN@github.com/org/repo.git@main

works as expected. However, trying to use this in Cloud Composer tells me the following:

PyPI package name must follow the format of 'identifier' specified in PEP-508. (https://peps.python.org/pep-0508/)

I'm not sure what I should do here. Does anyone have some experience with this?

Vinay Keerthi
  • 31
  • 3

1 Answers1

3

If the repo hosting your private Python packages accepts external calls, you can generate a pip.conf file with params to access to this repo :

[global]
extra-index-url = https://username:password@yourrepo.com/simple

Then you have to copy this file to Cloud Composer bucket : {composer_bucket_name}/config/pip/pip.conf

You can also host your Python private packages in GCP Artifact registry, then generate a pip.conf file allowed to access to the created Python repo in Artifact registry and then copy this file to the Composer bucket : {composer_bucket_name}/config/pip/pip.conf

After this you can add your private Python Packages in the PyPi packages list from Cloud Composer, like public packages :

Composer PyPi Packages

You can also check this link from official documentation : https://cloud.google.com/composer/docs/how-to/using/installing-python-dependencies

Mazlum Tosun
  • 5,761
  • 1
  • 9
  • 23