I need help with a little web scraper I have written and put into my pythonanywhere account to let it run several times per day.

here is my code:

import requests
from bs4 import BeautifulSoup
import time
import random

list_all_results = []


for i in range(1, 3):
    time.sleep(random.uniform(1.5, 2))

    print("Scraping page "   str(i)   "/745")

    try:
        URL = "https://www.futbin.com/players?page="   str(i)
        platform = "pc"
        cookies = {"platform": platform}
        page = requests.get(URL, cookies=cookies)

        soup = BeautifulSoup(page.content, "html.parser")

        result_names = soup.find_all("a", attrs={"class": "player_name_players_table"})
        result_ratings = soup.find_all(
            "span",
            attrs={"class": lambda r: r.startswith("form rating ut21") if r else False},
        )
        result_rarity = soup.find_all("td", {"class": "mobile-hide-table-col"})

        result_prices_pc = soup.find_all(
            "span", attrs={"class": "pc_color font-weight-bold"}
        )

        list_names = []
        list_ratings = []
        list_rarities = []
        list_prices = []

        for name in result_names:
            list_names.append(name.text)

        for rating in result_ratings:
            list_ratings.append(rating.text)

        for rarity in result_rarity:
            list_rarities.append(rarity.text)

        for price in result_prices_pc:
            n = price.text.strip()
            if "K" in n:
                n2 = n.replace("K", "")
                full_int = int(float(n2) * 1000)
                list_prices.append(full_int)
            elif "M" in n:
                n2 = n.replace("M", "")
                full_int = int(float(n2) * 1000000)
                list_prices.append(full_int)
            else:
                list_prices.append(int(price.text.strip()))

        int_list_length = len(list_names)
        for i in range(0, int_list_length):
            list_all_results.append(
                tuple(
                    (list_names[i], list_ratings[i], list_rarities[i], list_prices[i])
                )
            )

        with open("/home/exec85/scrape/pc.txt", "a") as f:  # create new .txt file and write content to file
            f.write(f"{list_all_results}")



    except:
        pass

print("FINISHED")

For some reason I dont get any result printed, so I assume nothing gets scraped and also the .txt file is not created. Even if I manually create the .txt file it gets not filled.

Running the script on my local machine all works fine.

CodePudding user response：

Your code works well but if you want the code works on PythonAnywhere, you need to have a paid account but you can reach this site list.