Pip install Packages on AWS Lambda and Store them on AWS S3-CodePudding

I am trying to reduce time on installing python packages on AWS Lambda.

For now, my codes are

# pip install custom package to /tmp/ and add to path
for package in ['requests==2.25.1', 'xlrd==2.0.1', 'pandas==1.2.4', 'numpy==1.19.5']:
    t1 = time.time()
    subprocess.call(f'pip install {package} -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    t2 = time.time()
    print(f'Install {package} in {round(t2-t1,2)} seconds')
sys.path.insert(1, '/tmp/')

but numpy and pandas take too long to install (~120s for pandas and ~50s for numpy).

I think this is a big charge if my function is called too frequently.

So I think that if I can install them into temp folder and push to S3, I can download and add to Lambda layer to call them without installing again.

I try zipping all temp directory but Lambda prevents me to writing files. Is there any way to overcome this?

Thank you for your reading!

CodePudding user response：

What i understand from the problem is that you need way to use the libs(numpy and pandas) in your lambda function.

One way to dealing such situation is to install all your libs under the one directory

pip install -r panda -t python/lib/pythonx.x/site-packages/

zip and upload it as a lambda layer. After that attach this layer to the lambda function, in that way you dont have to deal with installation of libraries inside your actual function.

There is really good article which explains how to create the lambda layer.

lambda layer - help

CodePudding user response：

If you want to use those packages you must install them in a folder with the code before deployment. In your lambda folder try

pip install --target ./package requests==2.25.1

And repeat with the rest of libraries. Then deploy your lambda with this folder as usual.

You can found a full example in AWS documentation.

https://docs.aws.amazon.com/lambda/latest/dg/python-package.html