Migrate Python3 project into AWS-CodePudding

I have been investigating how to migrate my python3 project into AWS but I am unable to find a solution. My python3 project is quite simple:

daemon.py -> This is an infinite while True loop. It does GETs requests to an API. It is important to make as much requests per sec as possible. Then it stores it into mysql db. The request is always the same, GET /api/check_stock.

while True:
  r = requests.post(url,json=body, proxies=proxy_dict)
  r_json = json.loads(r.content)
  insert_db(r_json)

ship.py -> Another while True loop looking for changes in the mysql, when detects a change sends a GET request to an API.

I found out about lambda but I am not sure if is a good idea to run infinite loop on it as I understand they charge you by the execution time... and this is infinite loop.

Thanks in advance.

CodePudding user response：

Whether it's effective for you depends only on how much you value each of those downloads and we don't have enough information to know the answer.

There's also a question of whether you want to run this all the time, or at some specific times for a specific duration. In the first case, fargate or ec2 may be better candidates, in the second, it depends how long you run the process for.

There are some things you can improve though with the current system - you seem to be waiting for each of the responses which means a lot of idle time. You could improve the throughput with either async or threadpool approaches. (unless it's really only a single endpoint listing all the items)

Another one is that you don't necessarily need to run ship.py as a monitoring loop if the state doesn't change very often. If you're already in AWS, you could do the notification as SQS messages instead, which would fire the ship.py process as a lambda - that's actually likely to save you some resources.

CodePudding user response：

Ok, found out best approach is to run daemon in EC2 with Aurora and try to migrate ship.py as viraptor said.

CodePudding user response：

if you do use a lambda serverless python project, you will be to do what you intend but you will billed quite substantially.

here are the docs: AWS-Lambda-Python

Also use SQS to push data from your script to the database.