0

I'm trying to submit the following toy snippet as a job in GCP ml-engine:

import tensorflow as tf
import numpy as np
import scipy.misc

x = np.zeros([10, 10, 1])
y = scipy.misc.imresize(x[:, :, 0], [50, 50, 1], interp='nearest')
print(y)
print(y.shape)

It is giving the following error after the job starts on server:

File "/root/.local/lib/python2.7/site-packages/teste/test.py", line 6, in <module>
y = scipy.misc.imresize(x[:, :, 0], [50, 50, 1], interp='nearest')
AttributeError: 'module' object has no attribute 'imresize'

It works perfectly on local, and according to Cloud-ML docs, the scipy package is supported. Apparently it is not an issue with the module itself, as the import statement is not giving any error.

Willian Mitsuda
  • 1,249
  • 1
  • 12
  • 14
  • When you use ml-engine, you have to build a package using a setup.py file. Try adding a dependency to scipy (using the 'install_requires' field of the setup function). – ma3oun Apr 17 '17 at 14:16
  • 1
    Actually just adding scipy is not enough, you have to add pillow as the accepted answer stated. – Willian Mitsuda Apr 17 '17 at 14:39

2 Answers2

3

scipy.misc.imresize requires PIL to be installed, which you probably have installed locally (since it works).

To ensure your code runs correctly in the cloud, you need to ensure pillow is installed. If you have created your own setup.py include pillow in the list of requirements. If you have to create your own, create a setup.py like this:

from setuptools import find_packages
from setuptools import setup

REQUIRED_PACKAGES = ['pillow']

setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My trainer application package.'
)

(source, with one important modification, the packages attribute)

See the official CloudML Engine documentation for more information about recommended directory layout and packaging instructions.

rhaertel80
  • 8,254
  • 1
  • 31
  • 47
  • It is curious that it is not required to add scipy to install_requires to use some of its functions, I tried calling scipy.misc.factorial without adding scipy and it was working. – Willian Mitsuda Apr 17 '17 at 14:42
  • Apparently it is not clear in the documentation which modules are implicitly made available and which requires a explicit install_requires, even though they are stated as available in the runtime image – Willian Mitsuda Apr 17 '17 at 14:45
  • 1
    This case is particularly confusing because the image manipulation functions in scipy.misc require PIL, but scipy doesn't install PIL. The CloudML environment does include scipy but not PIL. For a list of what is already installed, see https://cloud.google.com/ml-engine/docs/concepts/runtime-version-list – rhaertel80 Apr 17 '17 at 14:52
1

It was deprecated in 1.3.0. Contrary to using Pillow, reinstall scipy 1.0.0

pip install scipy==1.0.0

Or

pip3 install scipy==1.0.0
Karan Shah
  • 417
  • 6
  • 21