Webscraping Python BeautifulSoup-CodePudding

I am coding a python code for fun which will find the lowest train ticket price in Eurostar. I am very new to BeautifoulSoup so I do not know much about it. For some reason, the code does not retrieve information from the "ul" tables when theoretically it should.

The code is the following:

input_parser = InputParser()
input_parser.inputDestinations("London","Paris")
input_parser.adults=2
input_parser.inputDates("2021-10-08","2021-10-10")

URL = input_parser.createURL()
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

results = soup.find_all("ul", {"class": "train-table"})

The class input parser basically returns the URL based on the particular data:

class InputParser():

    def __init__(self):
        self.mapOfDestinations = {"London": "7015400", "Paris": "8727100", "Brussels": "8814001"}
        self.destinations = []
        self.adults = 0
        self.departureDate = ""
        self.arrivalDate = ""

    def inputDestinations(self, departureDestination, arrivalDestination):
        self.destinations.append(self.mapOfDestinations[departureDestination])
        self.destinations.append(self.mapOfDestinations[arrivalDestination])

    def inputDates(self, departureDate, arrivalDate):
        self.departureDate = departureDate
        self.arrivalDate = arrivalDate

    def inputAdults(self, numberOfAdults):
        self.adults = numberOfAdults

    def createURL(self):
        default_URL = "https://booking.eurostar.com/uk-en/train-search?origin={0}&destination={1}&adult={2}&outbound-date={3}&inbound-date={4}". \
            format(self.destinations[0], self.destinations[1], self.adults, self.departureDate, self.arrivalDate)
        return default_URL

My code should return the "ul" table linked to the "train-table" but it returns None. Any idea what I am doing wrong?

If you want to look at the source code the code gives the following URL: https://booking.eurostar.com/uk-en/train-search?origin=7015400&destination=8727100&adult=1&outbound-date=2021-10-08&inbound-date=2021-10-10

Thank you so much!

CodePudding user response：

The data you see is loaded from external URL so beautifulsoup doesn't see it. But you can use requests module to simulate this query:

import json
import requests

origin = "7015400"
destination = "8727100"

api_url = f"https://api.prod.eurostar.com/bpa/train-search/uk-en/{origin}/{destination}"
params = {
    "outbound-date": "2021-10-08",
    "inbound-date": "2021-10-10",
    "adult": "1",
    "booking-type": "standard",
}

headers = {"X-apikey": "0aa3d4b7e805493c8e310cfb871c4344"}

data = requests.get(api_url, params=params, headers=headers).json()

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

for j in data["outbound"]["journey"]:
    for c in j["class"]:
        if "price" in c:
            print(
                "{:<10} {:<10} {:<10} {:<10}".format(
                    j["departureTime"],
                    j["arrivalTime"],
                    c["remaining"],
                    c["price"]["adult"],
                )
            )

Prints:

07:01      10:17      150        134.5     
07:01      10:17      20         149.5     
07:01      10:17      47         245       
08:01      11:17      70         134.5     
08:01      11:17      2          179.5     
08:01      11:17      30         245       
10:24      13:47      27         134.5     
10:24      13:47      10         179.5     
10:24      13:47      31         245       
12:24      15:47      70         134.5     
12:24      15:47      50         219.5     
12:24      15:47      13         245       
16:31      19:47      7          134.5     
16:31      19:47      41         219.5     
16:31      19:47      31         245       
19:01      22:17      45         134.5     
19:01      22:17      8          149.5     
19:01      22:17      42         245       
20:01      23:17      35         74.5      
20:01      23:17      19         119.5     
20:01      23:17      51         245