I have a lambda that triggers off an S3 bucket upload (it basically converts a PDF to a dataframe and writes it to a different s3 bucket). Both of these belong to AWS account A. I would like to allow cross-account s3 access to trigger this lambda from another IAM user from account B (Administrator), however I am having issues with the GetObject operation. Here is how my lambda in account A looks:
LOGGER = logging.getLogger(__name__)
logging.basicConfig(level=logging.ERROR)
logging.getLogger(__name__).setLevel(logging.DEBUG)
session = boto3.Session(
aws_access_key_id="XXXX",
aws_secret_access_key="XXXX",
)
s3 = session.resource('s3')
dest_bucket = 'bucket-output'
csv_buffer = StringIO()
def lambda_handler(event,context):
source_bucket = event['Records'][0]['s3']['bucket']['name']
pdf_name = event['Records'][0]['s3']['object']['key']
LOGGER.info('Reading {} from {}'.format(pdf_name, source_bucket))
pdf_obj = s3.Object(source_bucket,pdf_name)
fs = pdf_obj.get()['Body'].read() #### code is failing here
df = convert_bytes_to_df(BytesIO(fs))
df.to_csv(csv_buffer,index=False)
s3.Object(dest_bucket,str(pdf_name.split('.')[0]) ".csv").put(Body=csv_buffer.getvalue())
LOGGER.info('Successfully converted {} from {} to {}'.format(pdf_name,source_bucket,dest_bucket))
The lambda is failing with this error:
ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
I'm aware it's bad practice to have keys in the lambda file but I can't change things at the moment.
The process works fine if I am uploading to the S3 bucket from within an IAM User in account A itself, but when I expose the S3 buckets to an IAM user from a separate account, the issues above start happening. This is the S3 bucket policy (terraform) allowing cross-account access to an IAM user from account B:
resource "aws_s3_bucket_policy" "cross_account_input_access" {
bucket = aws_s3_bucket.statements_input.id
policy = <<EOF
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXX:user/Administrator"
},
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::capsphere-input"
]
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::XXXXXXXXX:user/Administrator"
},
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
}
]
}
And here is the policy attached to an IAM user from another AWS account B which enables Administrator from account B to write a pdf to account A's s3 bucket programmatically:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::bucket-name"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
}
]
}
I write the file to the bucket from Administrator using aws cli like this:
aws s3 cp filename.pdf s3://bucket-name
I can't figure out what else needs to change.
CodePudding user response：
What if you add Region information to the session:
session = boto3.Session(
aws_access_key_id="XXXX",
aws_secret_access_key="XXXX",
region_name="<REGION>"
)
CodePudding user response：
It appears that your situation is:
Account A contains:
- An AWS Lambda function
- A 'source' bucket used to trigger the Lambda function
- A 'destination' bucket used by the Lambda function to store output
You want to allow the Administrator IAM User in Account B to upload a file to the source bucket in Account A. This user should be able to retrieve the output from the destination bucket in Account A.
The following would achieve these goals:
- Create an IAM Role in Account A and associate it with the Lambda function. Assign permissions to allow
GetObjectfrom the source bucket andPutObjectto the destination bucket. There should be no need to reference any credentials within the Lambda function itself, since any necessary permissions will be provided via this IAM Role. The policy would be:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject"
"Resource": "arn:aws:s3:::source-bucket/*"
},
{
"Effect": "Allow",
"Action": "s3:PutObject"
"Resource": "arn:aws:s3:::destination-bucket/*"
}
]
}
- Add a Bucket Policy on the source bucket that permits the
Administratoruser in Account B toPutObjectinto the bucket:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": "s3:PutObject",
"Resource": "arn:aws:s3:::source-bucket/*"
}
]
}
- Add a Bucket policy on the destination bucket that permits the
Administratoruser in Account B toGetObjectfrom the bucket and, I presume, list the bucket and delete objects that have been processed:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::destination-bucket"
},
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::Account-B:user/Administrator"
},
"Action": [
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": "arn:aws:s3:::destination-bucket/*"
}
]
}
- Since this is cross-account access, permission must also be granted to the
AdministratorIAM User in Account B to let them access the source and destination buckets. This policy is not required if they already have a lot of S3 permissions, such ass3:*, since it would work on any buckets including buckets in different AWS Accounts. This policy would go on the IAM User:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::destination-bucket"
]
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:DeleteObject"
],
"Resource": [
"arn:aws:s3:::source-bucket/*",
"arn:aws:s3:::destination-bucket/*"
]
}
]
}