I can't move forward more than one page with Selenium-CodePudding

I'm triying to scrape a transfermark web and I can't move fordward more than one page. This is the url of website:

With this code I only can enter to sencond page. I'd like can navigate until the final.

Thanks

CodePudding user response：

May I suggest using requests with a custom header instead of Selenium?

The following code works:

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': '...'}
for x in range(1, 18):
    r = requests.get(f'https://www.transfermarkt.es/transfers/transfertagedetail/statistik/top/land_id_zu/0/land_id_ab/0/leihe//datum/2022-07-06/sort//page/{x}', headers=headers)
    soup = BeautifulSoup(r.text, 'html.parser')
    ### find the data you need, save it etc

You can find user agents for different browsers easy, for instance here: https://www.whatismybrowser.com/guides/the-latest-user-agent/chrome