We have a Python3 API created with the FastAPI framework that lives in a Lambda Function. The entire API and utility functions all live in a single API. I know that a better practice would be to break it into several smaller Lambda functions more along the lines of each route being in it's own function, but this is where it is at now.
We would like to re-architect this API so that it is not all deployed on a single Lambda, along with other changes, but we need to do some benchmarking in order to see what improvements or regressions in performance are gained by the changes as we make them.
In order to facilitate this benchmarking I have added some OpenTelemetry packages to add instrumentation that we can use with AWS Xray. The issue is that adding these packages to a layer push it over the 250MB size limit.
I have tried reducing package size by using CFLAGS and some pip options and it works in reducing the size enough, but then it doesn't work when deployed. It fails with the following error in CloudWatch
[ERROR] Runtime.ImportModuleError: Unable to import module 'app': libc.musl-x86_64.so.1: cannot open shared object file: No such file or directory
I followed this tutorial for reducing package size and I used the following command to build the dependency layer in the CI/CD pipeline
CFLAGS="-g0 -Wl,--strip-all -I/usr/include:/usr/local/include -L/usr/lib:/usr/local/lib" pip3 install -r requirements.txt -t layer/python --no-cache-dir --compile --global-option=build_ext --global-option='-j 4'
My requirements.txt file looks like this
internal-team-package-1
internal-team-package-2
boto3 >= 1.14, < 2.0
botocore >= 1.15, < 2.0
fastapi >= 0.60, < 1.0
pydantic >= 1.6, < 2.0
packaging >= 21.0, < 22.0
mangum >= 0.9, < 1.0
python-jose >= 3.2, < 4.0
requests >= 2.22, < 3.0
ulid-py >= 1.1, < 2.0
jsonmerge == 1.8
opentelemetry-sdk-extension-aws~=2.0
opentelemetry-propagator-aws-xray~=1.0
opentelemetry-exporter-otlp~=1.5
opentelemetry-instrumentation-fastapi==0.30b0
opentelemetry-instrumentation-botocore==0.30b0
opentelemetry-api
opentelemetry-sdk
The 2 internal packages are fairly small and they don't have an dependencies that are not in this requirements file.
Is this just a matter of removing one of these flags from the command? I have tried removing some of the flags, but I really don't understand them and reading up on them hasn't helped, so I am dropping flags almost randomly and haven't seen a fix.
I might just containerize the API in order to get the tracing, as it is a route that I was considering for improvements to the API, but I don't want the scope of my work to creep too much and would like to containerize after adding the instrumentation.
Additional info:
- Lambda is deployed via AWS CDK via a Gitlab Pipeline
- All of the deps in the requirements file are required
