I am designing a small sports news app as a school project that scrapes data from the web, posts it to a firebase realtime database and is then used in an application being built on android studio by my project partner. So far during development I have just been deleting the database and rebuilding it every time i run the code to prevent build-up of the same data. I am wondering how i would go about checking to see if a piece of data exists before it push the data to the database. Thanks if anyone is able to point me in the right direction. Here is my code for pushing the data to firebase:

ref = db.reference('/news')
ref.delete()

url = 'https://news.sky.com/topic/premier-league-3810'
content = requests.get(url)
soup = BeautifulSoup(content.text, "html.parser")
body = soup.find_all("h3", "sdc-site-tile__headline")

titles_list = []
links_list = []
for item in body:
    headline = item.find_all('span', class_='sdc-site-tile__headline-text')[0].text
    titles_list.append(headline)

    link = item.find('a', class_='sdc-site-tile__headline-link').get('href')
    links_list.append('https://news.sky.com'    link)

i=0
while i < len(titles_list):
    ref.push({
        'Title' : titles_list[i],
        'Link' : links_list[i] 
    })
    i =1

CodePudding user response：

There are a few main options here:

1. You can use a query to check if the data already exists, before writing it. Then when it doesn't exist yet, you can write it.

If multiple users/devices can be adding data at the same time, the above won't work as somebody may write their data just after you have checked if the values already exist. In that case you'll want to:

2. Use the values that are to be unique as the key of the data (so using child("your_unique_values").set instead of push), and use a transaction to ensure you don't overwrite each other's data.