EDIT 3:
So the problem may likely be in the set-up and configuration of my Lambda Layer Dependencies. I have a /bin directory containing 3 files:
- lambdazip.sh
- pdftk
- libgcj.so.10
pdftk is a pdf library, and libgcj is a dependency for PDFtk.
lambdazip.sh seems to set & modify PATH Variables.
I have tried uploading all 3 as 1 lambda layer.
I have tried uploading all 3 as 3 separate lambda layers.
I have not tried customizing the .zip file names, I know sometimes the Lambda Layer wants you to name the .zip file a specific name dependent on the language.
I have not tried customizing the "compatible architectures" & "compatible runtime" lambda layer settings.
EDIT 2:
I tried renaming the Lambda Layer as Python.zip because I heard that sometimes you need a specific naming convention for the Lambda Layer to work correctly. This also failed & produced the same error.
EDIT:
I have tried pulling the .py files out of the /surveys directory, so when they are zipped, they are in the root folder, but I still receive the same error: Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'surveys
Which files do I need to zip? Do I need to move certain files to the root?
I learned that I had accidentally zipped the directory which commonly caused this error.
I needed to zip the contents of the directory, which is a common solution.
Unfortunately this did not work for me.
I have a Lambda Function, and the code I have uploaded is a zipped folder of my /Archive directory.
From what I understand, many of the people who run into this "[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function':" have issues because of their Lambda Handler.
My Lambda handler is: lambda_function.lambda_handler so this doesn't appear to be my issue.
Another common problem I've noticed on Stackoverflow, appears to be with how people are compressing & zipping the files they upload to the Lambda Function.
Do I need to move my lambda_function.py? Sometimes this CloudWatch error occurs because the lambda_function.py is not in the ROOT directory.
Does my survey directory need to move?
I think the folders & directories I have here may be causing my issue.
Do I need to zip the directories individually?
Can I resolve this error by Zipping the entire project?
For more information, I also have a Lambda Layer for PDF Toolkit, called pyPDFtk in the codebase. In that Lambda layer is a zipped /bin with binaries inside.
If there is anything I can alter/change within my code or AWS configuration, please let me know, and I can return new CloudWatch error logs for you.
lambda_function.py
"""
cl_boost-pdfgen manages form to
pdf merge and mail
"""
import json, base64
import os, sys
from string import Template
from boost import PageCalc, AwsWrapper
from boost.tools import helper
from boost.surveys import ALLOWED_SURVEYS
os.environ['LAMBDA_TASK_ROOT'] = os.environ['LAMBDA_TASK_ROOT'] if 'LAMBDA_TASK_ROOT' in os.environ else '/usr/local'
os.environ['PDFTK_PATH'] = os.environ['LAMBDA_TASK_ROOT'] + '/bin/pdftk'
os.environ['LD_LIBRARY_PATH'] = os.environ['LAMBDA_TASK_ROOT'] + '/bin'
# must import after setting env vars for pdftk
import pypdftk
# Constants
BUCKET_NAME = os.environ['BUCKET_NAME'] if 'BUCKET_NAME' in os.environ else 'cl-boost-us-east-1-np'
RAW_MESSAGE = Template(b'''From: ${from}
To: ${to}
Subject: MySteadyMind Survey results for ${subjname}
MIME-Version: 1.0
Content-type: Multipart/Mixed; boundary = "NextPart"
--NextPart
Content-Type: multipart/alternative; boundary="AlternativeBoundaryString"
--AlternativeBoundaryString
Content-Type: text/plain;charset="utf-8"
Content-Transfer-Encoding: quoted-printable
See attachment for MySteadyMind report on ${subjname}
--AlternativeBoundaryString
Content-Type: text/html;charset="utf-8"
Content-Transfer-Encoding: quoted-printable
<html>
<body>=0D
<p>See attachment for MySteadyMind Report on </b> ${subjname} </b>.</p>=0D
</body>=0D
</html>=0D
--AlternativeBoundaryString--
--NextPart
Content-type: application / pdf
Content-Type: application/pdf;name="${filename}"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;filename="${filename}"
${pdfdata}
--NextPart--''')
EMAIL_TAKER = True
#DEFAULT_EMAIL = os.environ['DEFAULT_EMAIL'] if 'DEFAULT_EMAIL' in os.environ else 'support@mysteadymind.com'
DEFAULT_EMAIL = os.environ['DEFAULT_EMAIL'] if 'DEFAULT_EMAIL' in os.environ else 'marshall@volociti.com'
SUBJECT = 'Evaluation for %s'
NAME_PATH = ['Entry', 'Name']
#EXTRA_EMAILS = os.environ['EXTRA_EMAILS'].split(",") if 'EXTRA_EMAILS' in os.environ else ['seth@mysteadymind.com']
EXTRA_EMAILS = os.environ['EXTRA_EMAILS'].split(",") if 'EXTRA_EMAILS' in os.environ else ['marshall@volociti.com']
# Lambda response
def respond(err, res=None):
"""
parameters are expected to either be
None or valid JSON ready to go.
:param err:
:param res:
:return:
"""
return {
'statusCode': '400' if err else '200',
'body': err if err else res,
'headers': {
'Content-Type': 'application/json',
},
}
def check_basic_auth(headers):
"""
pull out the auth header and validate.
:param headers:
:return:
# Retrieve values from env
vid = os.environ['uid']
vpw = os.environ['pwd']
encoded = "Basic " + base64.b64encode("%s:%s" % (vid,vpw))
# compare
return headers['Authorization'] == encoded
"""
return True
def lambda_handler(event, context):
"""
receive JSON, produce PDF, save to S3,
email via SES.... bring it!
"""
err = None
rsp = None
#Have to none out addresses for future lambda runs to not cause issues with appending.
ADDRESSES = None
ADDRESSES = {'from': "marshall@volociti.com",
'to': [DEFAULT_EMAIL] + EXTRA_EMAILS}
"""
ADDRESSES = {'from': "support@mysteadymind.com",
'to': [DEFAULT_EMAIL] + EXTRA_EMAILS}
"""
# check auth
if not check_basic_auth(event['headers']):
print ("Failed to authenticate")
return False
# get dataq
data = json.loads(event['body'])
# make sure its legit
if (data['Form']['InternalName'] not in ALLOWED_SURVEYS):
return False
# read in template and prep survey type and scoreit
pcalc = PageCalc(data, data['Form']['InternalName'])
pcalc.score()
pcalc.flat['Name'] = data['Section']['FirstName'] + \
" " + data['Section']['LastName']
# output pdf to temp space
# baseName = str(data['Entry']['Number']) + "-" + pcalc.survey + "-" + \
# data['Section']['LastName'].replace(' ','') + ".pdf"
baseName = str(data['Entry']['Number']) + "-MySteadyMind-" + \
data['Section']['LastName'].replace(' ','') + ".pdf"
filename = "/tmp/" + baseName
pypdftk.fill_form(pcalc.pdf_path, pcalc.flat, out_file=filename)
# -- Post Processing after PDF Generation -- #
# fetch the client wrapper(s)
aws = AwsWrapper()
# get PDF data and prep for email
try:
# save the pdf to S3
print("save %s to S3" % filename)
aws.save_file(BUCKET_NAME, pcalc.survey,filename)
# read in the pdf file data and
# base64 encode it
buff = open(filename, "rb")
pdfdata = base64.b64encode(buff.read())
buff.close()
ADDRESSES['to'].append(data['Section']['Email']) if EMAIL_TAKER else None
# gather data needed for email body
data = {"from": ADDRESSES['from'],
"to": ', '.join(ADDRESSES['to']),
"subjname": pcalc.flat["Name"],
"filename": baseName,
"pdfdata": pdfdata
}
print("sending email via SES to: %s" % ', '.join(ADDRESSES['to']))
# build MMM email and send via SES
response = aws.send_raw_mail(ADDRESSES['from'],
ADDRESSES['to'],
RAW_MESSAGE.substitute(data))
# send JSON response
rsp = '{"Code": 200, "Message": "%s"}' % response
except Exception as ex:
# error trap for all occassions
errmsg = "Exception Caught: %s" % ex
# notify local log
print(errmsg)
# and lambda response
err = '{"Code":500, "Message":"%s"}' % errmsg
# done
return respond(err, rsp)
if __name__ == '__main__':
# use this option to manually generate from raw csv of cognitoforms
if len(sys.argv) > 1:
import csv
with open(sys.argv[1], 'rU') as csvfile:
csvreader = csv.DictReader(csvfile, delimiter=',',)
for row in csvreader:
jsondata = helper.create_json(row)
fakeevent = {'body': json.dumps(jsondata), "headers": []}
lambda_handler(fakeevent, None)
# use this option to manually generate from raw json webhook response from cognito in a dir called generated/
else:
rng = range(3,4)
print (rng)
print ("Attempting to parse files: " + str(rng))
for i in rng:
try:
print ('./generated/queue/' + str(i) + '.json')
f = open('./generated/queue/' + str(i) + '.json', 'r')
jsondata = f.read().replace('\n', '')
f.close()
#jdata = json.loads(jsondata)
fakeevent = {'body': jsondata, "headers": []}
lambda_handler(fakeevent, None)
except:
print ("error. file not found: " + str(i))
lambdazip.sh
#!/bin/bash
PYTHON_PATH=$VIRTUAL_ENV/..
BASE_PATH=$PWD
cd $VIRTUAL_ENV/lib/python3.9/site-packages/
zip -x "*pyc" -r9 $BASE_PATH/dist/cl-boost.zip pypdftk*
cd $BASE_PATH
zip -r $BASE_PATH/dist/cl-boost.zip bin
cd $BASE_PATH
zip -x "*pyc" -r9 $BASE_PATH/dist/cl-boost.zip boost* pdf_surveys
