I've come to a point where I kind of feel I made the wrong choice with choosing Vapor as the platform for a eCommerce API - not sure.

On key part for my API is to fetch products feed from a supplier API that comes in a json format and the size of this feed is around 120 MB. Ideally I'd like the "sync" process to be a queued Job (even a Batch of Jobs - feed divided into chunks, for each product model containing it's variations) but the problem overall is that I get 413 Request Entity Too Large from DynamoDB.

{
    "message": "Error executing \"PutItem\" on \"https://dynamodb.eu-north-1.amazonaws.com\"; AWS HTTP error: Client error: `POST https://dynamodb.eu-north-1.amazonaws.com` resulted in a `413 Request Entity Too Large` response Unable to parse error information from response - Error parsing JSON: Syntax error",
    "context": {
        "exception": {
            "class": "Aws\\DynamoDb\\Exception\\DynamoDbException",
            "message": "Error executing \"PutItem\" on \"https://dynamodb.eu-north-1.amazonaws.com\"; AWS HTTP error: Client error: `POST https://dynamodb.eu-north-1.amazonaws.com` resulted in a `413 Request Entity Too Large` response Unable to parse error information from response - Error parsing JSON: Syntax error",
            "code": 0,
            "file": "/var/task/vendor/aws/aws-sdk-php/src/WrappedHttpHandler.php:195",
            "previous": {
                "class": "GuzzleHttp\\Exception\\ClientException",
                "message": "Client error: `POST https://dynamodb.eu-north-1.amazonaws.com` resulted in a `413 Request Entity Too Large` response",
                "code": 413,
                "file": "/var/task/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:113"
            }
        },
        "aws_request_id": "452bd124-548c-58a2-ac12-8a41e422e6a7"
    },
    "level": 400,
    "level_name": "ERROR",
    "channel": "staging",
    "datetime": "2022-08-04T11:33:57.191435 00:00",
    "extra": {}
}

Vapor docs state:

Due to AWS Lambda limitations, file uploads made directly to your application backend can only be up to roughly 4.5MB in size. This is a hard limit imposed by AWS

A source from Google states:

There is a hard limit of 6mb when it comes to AWS Lambda payload size. This means we cannot send more than 6mb of data to AWS Lambda in a single request.

Developers will typically run into this limit if their application was using AWS Lambda as the middle man between their client and their AWS S3 asset storage.

Keeping this in mind it makes sense that the HTTP request fetching the 120 MB json feed "is too large" for Lambda to handle.

Also I found a source saying queued Job total payload size is limited to 256 kb - can't find the source anymore tho.

Any suggestions on how to tackle this?

CodePudding user response：

DynamoDB has a 400kb per item limit. You're trying to PutItem an item that's too large. See item limits in the docs.

To get around this, you can break up your items into small pieces (I'd need to see the schema to give any suggestions here). You can also upload the full 120MB blob to S3 and then store only the metadata you need to rapidly query pointer to S3 inside of DynamoDB.