How to upload to s3 from ec2 log files through lambda & boto3-CodePudding

I want to upload only Ec2 log files through lambda.

I have code for this

import json
import os
import datetime as dt
import boto3
import socket

region = 'ap-northeast-2'
instances = []
ec2_r = boto3.resource('ec2')
ec2 = boto3.client('ec2', region_name=region)

for instance in ec2_r.instances.all():
    for tag in instance.tags:
        if tag['Key'] == 'Log-Archive':
            if tag['Value'] == 'True':
                instances.append(instance.id)
             
def lambda_handler(event, context):
    s3 = boto3.client("s3")   
    dir_path = "/log"
    bucket = 'mybucket'
    x = dt.datetime.now()
    date = x.strftime("%Y%m%d")
    def log(in_path):
        for (dir_path, dir, files) in os.walk(in_path):
            for file in files:
                if date in file:
                    yield os.path.join(dir_path, file)
                    
    for file_name in log(dir_path):
        key = socket.gethostname()   '/'   file_name
        res = s3.upload_file(file_name, bucket, key)

    return {
        'statusCode': 200,
    }

When I tested this code it returned 200 code. but nothing in my bucket. And I tested this code on my local ec2 without lambda code, and It works

33

Response
{
  "statusCode": 200
}

Function Logs
START RequestId: 39f63d4f-e154-48b8-9fb3-51b98f927a30 Version: $LATEST
END RequestId: 39f63d4f-e154-48b8-9fb3-51b98f927a30
REPORT RequestId: 39f63d4f-e154-48b8-9fb3-51b98f927a30  Duration: 51.61 ms  Billed Duration: 52 ms  Memory Size: 128 MB Max Memory Used: 89 MB

Request ID
39f63d4f-e154-48b8-9fb3-51b98f927a30

Can I take some advices? thanks for all answers.

CodePudding user response：

Your Lambda will be running in a serverless compute environment, not in your EC2. That means the log directory that you are traversing is in the serverless Lambda environment, not the log directory in your EC2 instance. My guess is, the Lambda log directory contains no log files. Hence, the code ran successfully without uploading anything to S3.

My suggestion is to keep this python as a local script in the instance. Then, as you have already indicated, you can use AWS SSM to trigger this script.

CodePudding user response：

you probably miss to install in your ec2 the cloudwatch agent. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Install-CloudWatch-Agent.html Furthermore, there is no need to "move" log between services, because AWS provide automatic logging for everything, with Cloudwatch and Cloudtrail services