Adding 1 to a URL whilst scraping-CodePudding

If I have a url called youtube.com/user/1/ and I want the number to incrament to youtube.com/user/2/ whilst scraping the whole page, how would I do so? I've been trying to think of some ways to do this but I can't seem to find anything?

BeautifulSoup

It should work like this: youtube.com/user/1/ You scrape the page and then it moves to youtube.com/user/2/

If you could find the solution in beautiful soup that would be cool :) etc.

CodePudding user response：

while scraping:
    for i in range(amount_of_scrapes):
        scrape(f"youtube.com/user/{i 1}/")
    scraping = False

This is just an example but correct implemented it should work

CodePudding user response：

You could do something like

last_page_num = # set the last page number here

scraped_data = []
for page_num in range(1, last_page_num   1):
    url = f'youtube.com/user/{page_num}/'
    try:

        ################## SCRAPE URL ################## 
   
        ############ APPEND TO scraped_data ############

    except Exception as err: 
        print(f'unable to scrape {url}\n{str(err)}')
        # break # if you want to stop after first error

################# SAVE scraped_data #################