Can't build zip file for AWS lambda using Linux Alpine docker-CodePudding

I have Dockerfile based on Alpine linux that builds lambda.zip file for AWS Lambda. Here's Dockerfile:

FROM alpine:3.12

# -- Install OS packages:
RUN apk add gcc
RUN apk add --update --no-cache \
    bash \
    build-base \
    cargo \
    curl \
    docker \
    # gcc \
    git \
    g   \
    lftp \
    libc-dev \
    libffi-dev \
    libsodium-dev \
    libxslt-dev\
    libzmq \ 
    zeromq-dev \
    make \
    musl-dev \
    ncftp \
    nodejs \
    npm \
    openssh-client \
    openssl \
    openssl-dev \
    rsync \
    su-exec \
    tar \
    wget \
    zip 
# geos \ 
# geos-dev \
# libc-dev 

WORKDIR /tmp/

RUN echo "http://mirror.leaseweb.com/alpine/edge/community" >> /etc/apk/repositories
RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories
RUN apk add --virtual .build-deps \
    --repository http://dl-cdn.alpinelinux.org/alpine/edge/community \
    --repository http://dl-cdn.alpinelinux.org/alpine/edge/main \
    libc-dev geos-dev geos && \
    runDeps="$(scanelf --needed --nobanner --recursive /usr/local \
    | awk '{ gsub(/,/, "\nso:", $2); print "so:" $2 }' \
    | xargs -r apk info --installed \
    | sort -u)" && \
    apk add --virtual .rundeps $runDeps

RUN geos-config --cflags

# -- Install python:
RUN apk add --update --no-cache python3-dev python3 \
    && python3 -m ensurepip --upgrade \
    && pip3 install --upgrade pip pipenv setuptools docker-compose awscli shapely wheel \
    && rm -r /usr/lib/python*/ensurepip  \
    && if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi \
    && if [[ ! -e /usr/bin/python ]]; then ln

Here's script that works on docker Alpine image and builds lambda.zip that contains python files and all needed dependencies like boto3, pip, six etc.

 
#!/bin/bash

set -x
set -e

pipenv install
rm -rf lambda.zip

VENV=$(pipenv --venv)
TEMPDIR=$(mktemp -d)

rsync -r --progress ./* ${TEMPDIR}
rsync -r --progress ${VENV}/lib/python3.8/site-packages/* ${TEMPDIR}

pushd ${TEMPDIR}
zip -r lambda.zip ./*
popd

cp ${TEMPDIR}/lambda.zip ./

cleanup() {
    rm -rf ${TEMPDIR}
}

trap cleanup EXIT

Then I copy this lambda.zip to my local machine, unzip it and enable pipenv on my local machine to use only that packages in lambda.zip and i try to test Shapely package, for example test.py:

from shapely.geometry import Polygon
print(Polygon([[0, 0], [1, 0], [1, 1], [0, 1], [0, 0]]).minimum_clearance)

On AWS lambda and on my local machine I am getting same error with geos_c After:

Traceback (most recent call last):
  File "test.py", line 1, in <module>
    from shapely.geometry import Polygon
  File "/home/stark/code/job/lamba_python_312_stack/shapely/geometry/__init__.py", line 4, in <module>
    from .base import CAP_STYLE, JOIN_STYLE
  File "/home/stark/code/job/lamba_python_312_stack/shapely/geometry/base.py", line 19, in <module>
    from shapely.coords import CoordinateSequence
  File "/home/stark/code/job/lamba_python_312_stack/shapely/coords.py", line 8, in <module>
    from shapely.geos import lgeos
  File "/home/stark/code/job/lamba_python_312_stack/shapely/geos.py", line 87, in <module>
    _lgeos = load_dll('geos_c', fallbacks=alt_paths)
  File "/home/stark/code/job/lamba_python_312_stack/shapely/geos.py", line 60, in load_dll
    raise OSError(
OSError: Could not find lib geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so'].

If I unzip file, I got something like these:

@user# ls

iohttp
aiohttp-3.7.4.post0.dist-info
attr
attrs-21.2.0.dist-info
boto3
boto3-1.18.38.dist-info
botocore
cron_runner.py
setuptools-58.0.4.virtualenv
shapely
Shapely-1.7.1.dist-info
...
xmltodict.py
yarl
yarl-1.6.3.dist-info

I tried a lot of different solutions on Internet so you can see a mess in my Dockerfile, but they didn't work.

CodePudding user response：

You are installing some statically compiled dependencies in your Docker environment, like libc-dev, geos-dev, and geos. You have to also include those static dependencies in the Lambda deployment zip file. Also, to include statically compiled dependencies for use in AWS Lambda you have to use the same operating system Lambda uses, which is Amazon Linux, not Alpine Linux.

Luckily there are two alternatives now that make this much easier:

Lambda Layers

Lambda Layers are Lambda dependencies that can be packaged up in a reusable method, that can also be shared with other developers. In this case someone has already created a shapely Lambda Layer (and someone else here) that you can simply include in your Lambda function instead of trying to package it yourself.

If you still want to build it yourself you could look at that project's source code to see how they are building the layer.

Lambda Containers

Instead of creating a zip deployment, you could create a Docker image and deploy it to Lambda. You do have to implement a specific interface inside your Lambda container if you go this route, and it is easiest to do this by starting with one of the official AWS Lambda base images.