1

I am trying to create Python UDF in Amazon Redshift, and I have successfully created the UDF with no error. I have also created the required library for this UDF successfully. But when I execute the UDF, I get the error:

No Module Named pyffx. Please look at svl_udf_log for more information

I have downloaded the library from pypi.org and uploaded it to Amazon S3. This is the link I used to download the library:

https://pypi.org/project/pyffx/#files

create library pyffx
language plpythonu
from 's3://aws-bucket/tmp/python_module/pyffx-0.3.0.zip'
credentials
'aws_iam_role=iam role'
region 'us-east-1';

CREATE OR REPLACE FUNCTION schema.ffx(src VARCHAR)
RETURNS VARCHAR
STABLE
AS $$
    import pyffx
    src = unicode(src)
    value=(src)
    l=len(value)
    e = pyffx.String(b'secret-key', alphabet='abcedefghijklmnopqrstuvwxyz123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ', length=l)
    return e.encrypt(value)
$$ LANGUAGE plpythonu;
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
st_bones
  • 119
  • 1
  • 3
  • 12
  • Did the `create library pyffx` command execute without error? And, presumably, you put your actual IAM Role in the `credentials` line? Did you convert the `pyffx-0.3.0.tar.gz` file into a `.zip` format? – John Rotenstein Jul 01 '19 at 04:54
  • yes, create library pyffx executed without error. yes, i have put actual IAM Role in the credentials. yes, i converted the .tar.gz to .zip using below online convertor tool. https://www.zamzar.com/convert/tar-to-zip/ – st_bones Jul 01 '19 at 15:42

1 Answers1

0

I managed to get it to work... sort of.

I did the following:

  • Downloaded pyffx via the link you provided
  • Extracted the .tar.gz file and created a .zip of the files
  • Copied the .zip file to Amazon S3
  • Loaded the library using your CREATE LIBRARY command
  • Created the function

However, when I use the function, I receive the error:

Invalid operation: AttributeError: 'module' object has no attribute 'add_metaclass'

My research suggests that the six library (that provides Python 2 and 3 compatibility) is the source of this problem. The Python Language Support for UDFs - Amazon Redshift page indicates that six 1.3 is included in Redshift, yet Pip six.add_metaclass error says that this version does not include add_metaclass. The current version of six is 1.12.

I tried to include an updated six library in the code but wasn't successful. You might be able to wrangle better than me.

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • i uploaded six library to the redshift and import the six library in my code, i still get same error: "no module pyffx". CREATE OR REPLACE FUNCTION schema.ffx(src VARCHAR) RETURNS VARCHAR STABLE AS $$ import pyffx src = unicode(src) value=(src) l=len(value) e = pyffx.String(b'secret-key', alphabet='abcedefghijklmnopqrstuvwxyz123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ', length=l) return e.encrypt(value) $$ LANGUAGE plpythonu; – st_bones Jul 01 '19 at 16:09
  • How did you upload the `six` library? When I tried to `CREATE LIBRARY` for `six`, it complained that it already existed. Loading the library under a different name didn't change my error message. – John Rotenstein Jul 01 '19 at 23:16
  • I followed the same process i did for creating pyffx library. download the file and load to s3 and create the Six library. – st_bones Jul 03 '19 at 16:09
  • But it is not permissible to create a library with the same name as an existing library. Did you use `CREATE LIBRARY six`? – John Rotenstein Jul 03 '19 at 21:42
  • yes, I did CREATE LIBRARY SIX. i think issue is with the version of this package; pyffx is compatible with python 3, however Redshift is using python 2.7. what is your thoughts? – st_bones Jul 09 '19 at 10:40