I am trying to create project that

1. Scrape data from the website automatically in every 5 minutes and save into db
2. Flask app interacts with user. User requests data from db

python script to scrape data would be like:

import os
import time

while True:
  os.system(scrape.py)
  time.sleep(5*60)

I originally tried to scrape data from the website every time the user requests data. But I fount out that it works so slowly.

So now I am trying to make another python script that scrapes data from website and save them into db automatically. And flask app just needs to get data from db.

My question is, how can I run both python script and flask app at the same time? Or is there any better way to handle the problem?

CodePudding user response：

There is no reason for these two components to run in the same process, or even in the same application. They can share some modules (e.g. object definitions), but you need one batch process that builds your database, and one flask app that serves user requests. Keep them loosely coupled and you will have many benefits, including simplicity and the possibility of replacing one component with something else in the future.

So set up your project over several files, and create one execution point (file) for the flask app and one execution point for the scraper.

CodePudding user response：

With the use of multithreading, you can use a thread to handle scraping the data each five minutes and another thread to handle the flask app:

import threading
import os
import time
def scraping():
    while True:
        os.system(scrape.py)
        time.sleep(5*60)
def start_flask_app():
    #code
    pass
t1 = threading.Thread(target=scraping)
t2 = threading.Thread(target=start_flask_app)
t1.start()
t2.start()

You should wrap your flask app inside of a function and make it a target for a thread and then you can start that thread along with the scraping thread also.

For more information on multithreading in python (https://www.tutorialspoint.com/python/python_multithreading.htm)