1

I am using Google App Engine Flexible with a standard Python 3 runtime. I had been using pandas 1.0.5 until now. I want to start using pandas 1.3.4. So I updated requirements.txt and tried deploying to GAE. I got an error

Step #1: ERROR: Could not find a version that satisfies the requirement pandas==1.3.4 (from -r requirements.txt (line 16)) (from versions: 0.1, 0.2, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.4.3, 0.5.0, 0.6.0, 0.6.1, 0.7.0, 0.7.1, 0.7.2, 0.7.3, 0.8.0, 0.8.1, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1, 0.15.0, 0.15.1, 0.15.2, 0.16.0, 0.16.1, 0.16.2, 0.17.0, 0.17.1, 0.18.0, 0.18.1, 0.19.0, 0.19.1, 0.19.2, 0.20.0, 0.20.1, 0.20.2, 0.20.3, 0.21.0, 0.21.1, 0.22.0, 0.23.0, 0.23.1, 0.23.2, 0.23.3, 0.23.4, 0.24.0, 0.24.1, 0.24.2, 0.25.0, 0.25.1, 0.25.2, 0.25.3, 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5)
Step #1: ERROR: No matching distribution found for pandas==1.3.4 (from -r requirements.txt (line 16))
Step #1: WARNING: You are using pip version 20.2.2; however, version 21.3.1 is available.
Step #1: You should consider upgrading via the '/env/bin/python -m pip install --upgrade pip' command.
Step #1: The command '/bin/sh -c pip install -r requirements.txt' returned a non-zero code: 1
Finished Step #1
ERROR

I see that the latest pandas version it lists is 1.1.5, even though pandas 1.3.4 is available.

  1. How often does GCloud update the package versions that are available on its standard runtimes?

  2. Is the version availability related to the version of pip that is part of the custom runtime?

  3. Is there any way I can use the latest version of packages without having to build a custom runtime?

VRA
  • 91
  • 9
  • What version of Python are you using? Pandas 1.3.4 requires Python >=3.7.1. If your python doesn't meet the requirement, GAE might block the installation – NoCommandLine Nov 20 '21 at 01:51
  • My app.yaml contains `python_version: 3`. GCloud docs (https://cloud.google.com/appengine/docs/flexible/python/runtime) says that implies Python 3.7.2. – VRA Nov 20 '21 at 01:53
  • What happens when you try to run this on your local machine - do you get the same error about about Pandas? – NoCommandLine Nov 20 '21 at 01:55
  • The same `requirements.txt` file works fine. Local has Python 3.8.3. – VRA Nov 20 '21 at 02:01
  • Try setting ```runtime: python38``` in ```app.yaml`` – NoCommandLine Nov 20 '21 at 02:06
  • That didn't work. Got this: `ERROR: (gcloud.app.deploy) Your application does not satisfy all of the requirements for a runtime of type [python38]. Please correct the errors and try again.` Neither did `runtime: python37` though. – VRA Nov 20 '21 at 02:23
  • But that gave me an idea to play around with `python_version`. Tried this: `python_version: 3.8`. Got an error: `Step #0: ValueError: Invalid "python_version" field in "runtime_config" section of app.yaml: '3.8'. Valid options are: ['', '2', '3', '3.4', '3.5', '3.6', '3.7']` – VRA Nov 20 '21 at 02:24
  • Oh, `python_version: 3.7` worked! – VRA Nov 20 '21 at 02:27

1 Answers1

2

My original app.yaml had python_version: 3. The Google Cloud docs here state that python_version: 3 in app.yaml refers to Python 3.7.2. However, the build logs contained a line RUN virtualenv --no-download /env -p python3.6.

SOLUTION: Change the config to python_version: 3.7. Now the corresponding line in the build log says RUN virtualenv --no-download /env -p python3.7.

VRA
  • 91
  • 9
  • **QQ (quick question):** Why are you using App Engine Flexible? (Google recommends you use App Engine Standard for most use cases, and only use Flexible if you have special needs like custom runtimes, you have your own Docker container, etc.) App Engine Standard fully supports Pandas and doesn't suffer from the "weirdness" (nor long deployment times) you experienced with Flexible, as I answered in this other SO Q&A: https://stackoverflow.com/a/69820884/305689 – wescpy Jan 21 '22 at 01:48
  • **WRT (with regards to) the version numbers**: Here is what gets deployed for your specified `python_version`: a) Python _2.7.12_ as specified by **2** (default if `python_version` not supplied), b) Python _3.6.10_ as specified by **3**, c) Python _3.4.8_ as specified by **3.4**, d) Python _3.7.9_ as specified by **3.7**. Adding to my previous "QQ" comment: you can also use [Cloud Run](http://cloud.run) as another alternative to App Engine Flexible, esp. if you have a Docker container/`Dockerfile`. – wescpy Jan 21 '22 at 02:58
  • @wescpy 1. I don't, yet, need to customize the runtime. I also haven't containerized my app. However, I do need to be able to specify the CPU and RAM for the application. The specs for the [GAE Standard instances](https://cloud.google.com/appengine/docs/standard) are not sufficient for my application. That's why I'm having to use GAE Flexible. 2. Thanks for the update. Did you get these results from some official documentation or through your own experiments? Google's documentation page still lists only the 2 options I'd seen earlier i.e. (2->2.7.9; 3->3.7.2). – VRA Jan 24 '22 at 21:11
  • [@VRA](https://stackoverflow.com/users/6660567/vra): **1.** Makes sense. FWIW (for what it's worth), you can also specify [RAM](https://cloud.google.com/run/docs/configuring/memory-limits) and [CPU](https://cloud.google.com/run/docs/configuring/cpu) on Cloud Run which you can use _without_ containerizing your app (`Dockerfile` optional). **2.** For the available versions, I looked in these Flex deployment scripts http://github.com/GoogleCloudPlatform/python-runtime/tree/master/python-interpreter-builder/scripts because the Google documentation is wrong. I also filed a bug so they fix the docs. – wescpy Jan 29 '22 at 22:37
  • Naturally since I filed the bug, they assigned it to me. :P Anyway, the page is now updated to show all the correct versions deployed depending on the `python_version` setting: https://cloud.google.com/appengine/docs/flexible/python/runtime#interpreter I was also able to add a section on alternatives in case your app requires a version not available via these settings, including some of my Cloud Run migration work. Anyway, hope this docs update helps! – wescpy Feb 09 '22 at 02:18