How to grab on value from an s3 log using python?-CodePudding

I have created the connection to AWS s3 using Python. I have used the filter to get the type of log I need, but now I am stuck getting the specific value. I have tried creating a dictionary but does not parse well, so not quite sure if I should just convert to string and then parse the value that way or what is the best option.

I have created the connection with the login details.

s3_client = boto3.client('s3')
s3_bucket_name = 'halo-test-log-bucket'
s3 = boto3.resource('s3',
                    aws_access_key_id='XXXXXXXXXXXXXX',
                    
aws_secret_access_key='XXXXXXXXXXXXXXXXXXXXXXXXXXXXX')

This is where I get the fine:

my_bucket = s3.Bucket(s3_bucket_name)
for file in my_bucket.objects.filter(Prefix = ARCID '-quote'):
    file_name=file.key
    print(file_name)

My attempt at getting the value:

for file in bucket_list:
   obj = s3.Object(s3_bucket_name, file)
   data=obj.get()['Body'].read()
   data=str(data, 'utf-8')
   data=StringIO(data)
   ARCID_df=pd.read_csv(data, on_bad_lines='skip')

ARCID_df.iloc[[125]]

This does not work. Like I said before, was considering making this log into a string and then getting the specific value, but was hoping to get a dataframe or at least a dictionary.

Here is a sample of the log output:

    2022-08-26 18:50:01.002  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : productId:1111

2022-08-26 18:50:01.002  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : arcId:ARC667766666

2022-08-26 18:50:01.002  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : ##### calling getConfig  #######

2022-08-26 18:50:01.002  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : getConfig Req Obj:{"arcId":"ARC6667766666","productId":"1111"}

2022-08-26 18:50:01.031  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : getConfig ResponseObj:{

  "redisHost": "happy.com", 

  "redisPassword": "1111111111111111", 

  "redisPort": "11111", 

  "redisUser": "default", 

  "statusCode": "200"

}



2022-08-26 18:50:01.031  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : REDIS getConfig Response ConfigStatus:200

2022-08-26 18:50:01.031  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : Input Age:00

2022-08-26 18:50:01.031  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : Input Gender:1

2022-08-26 18:50:01.031  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : Input State:MD

2022-08-26 18:50:01.031  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : ### Input term:15

2022-08-26 18:50:01.034  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : fetched value from Redis arcid DIST:7777

2022-08-26 18:50:01.035  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : ##distId:7777

2022-08-26 18:50:01.035  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : ####### Input Validation and Data Preparation######## 

2022-08-26 18:50:01.035  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : ### Inside GetServiceApi Call

2022-08-26 18:50:01.035  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : [getService]getServiceUrl:http://getservice-utility-container-external.haven-life-utilities:8092/getServices

2022-08-26 18:50:01.035  INFO 1 --- [http-nio-8092-exec-8] c.afficiency.quote.service.QuoteService  : GetServiceApi Req Obj:{"inProductCode":"1111"}

Output:

2022-08-29 21:05:43.130 INFO 1 --- [http-nio-8092-exec-2] c.afficiency.quote.service.QuoteService : productId:9339
69                                       2022-08-29 21:05:43.235 INFO 1 --- [http-nio-...

CodePudding user response：

Hi everyone sorry for the uncertainty in my last bit of code. I did figure it out. What I did was get the output into a string and then split on the information before and after the value, I was looking for.

I first iterated over the logs to find the specific log I was looking for, then appended it to a list. Then I access the first file in the list, turn it into an object, and then read used .get() to read the body of the text, then read the bytes into a string.

Once I had it as a string, I split it into where the value was before and after.

bucket_list = []
my_bucket = s3.Bucket(s3_bucket_name)
for file in my_bucket.objects.filter(Prefix = ARCID '-quote'):
    file_name=file.key
    bucket_list.append(file_name)



for file in bucket_list:
        obj = s3.Object(s3_bucket_name, file)
        data_obj=obj.get()['Body'].read()
        data_str=data_obj.decode('utf-8')



rate_mult_key = data_str.split("Ans Option List:[")
rate_mult_value = rate_mult_key[2].split(']')
rate_multiplier = float(rate_mult_value[0])

Output:

1.95