Scraping Dynamic Webpages w/ a date selector-CodePudding

I am looking to use the requests module in python to scrape:

https://www.lines.com/betting/nba/odds

This site contains historical betting odds data.

The main issues, is there is a date selector on this page, and i can not seem to find where the date value is stored. Ive tried looking in the headers and the cookies, and still cant seem to find where date is stored, in order to programmatically change it, to scrape data from different dates.

Looking on the network tab, it seems like it is pulling this data from:

https://www.lines.com/betting/nba/odds/best-line?date=2023-01-23'

However, even with using the headers, i am unable to access this site. It just returns the data from:

https://www.lines.com/betting/nba/odds

which is the current date.

I am looking to do so without using a different method (i.e. Selenium) which seems pretty straight forward (Open Page -> Download Data -> Click Previous Date -> Repeat)

Here is my code to do so:

import requests
url = 'https://www.lines.com/betting/nba/odds/'
requests.get(url).text

Thanks!

CodePudding user response：

Try to pass headers={"X-Requested-With": "XMLHttpRequest"} to the request:

import requests
import pandas as pd
from itertools import cycle
from bs4 import BeautifulSoup

url = "https://www.lines.com/betting/nba/odds/best-line?date=2023-01-28"

soup = BeautifulSoup(
    requests.get(url, headers={"X-Requested-With": "XMLHttpRequest"}).content,
    "html.parser",
)

odds = []

for o in soup.select(".odds-list-col"):
    matches = [t["title"] for t in o.select(".odds-list-team")]
    teams = cycle(matches)

    for od in o.select(".odds-list-val"):
        odds.append(
            [
                next(teams),
                " vs ".join(matches),
                od.find_previous(class_="odds-col-title").text.strip(),
                od.get_text(strip=True, separator=" "),
            ]
        )

df = pd.DataFrame(odds, columns=["Team", "Match", "Odd", "Value"]).pivot(
    index=["Team", "Match"], columns="Odd", values="Value"
)
print(df)

Prints:

Odd                                      M/L            O/U          P/S
Team         Match                                                      
Bucks        Bucks vs Pacers            -325  o237.0 (-110)  -7.5 (-115)
Cavaliers    Cavaliers vs Thunder       -115              —  -1.0 (-110)
Grizzlies    Grizzlies vs Timberwolves  -150  o237.0 (-110)  -3.0 (-110)
Heat         Magic vs Heat              -275  u218.0 (-110)  -7.0 (-110)
Magic        Magic vs Heat               265  o218.0 (-110)   7.5 (-110)
Pacers       Bucks vs Pacers             280  u237.5 (-110)   8.0 (-108)
Raptors      Raptors vs Warriors         188              —   5.5 (-110)
Thunder      Cavaliers vs Thunder       -105              —   1.0 (-110)
Timberwolves Grizzlies vs Timberwolves   145  u237.5 (-110)   3.5 (-110)
Warriors     Raptors vs Warriors        -205              —  -5.0 (-114)