How to crawl events from calendar with Python and Beautiful Soup-CodePudding

I want to extract links of all events from the calendar on this website, and later to crawl info about each event.

This is the URL: https://thalheim.ch/index.php/aktuell/veranstaltungen

This is the code that I wrote:

from bs4 import BeautifulSoup
import requests

def get_website_news_links_thalheimCh():
        
    url = 'https://thalheim.ch/index.php/aktuell/veranstaltungen'
    
    response = requests.get(url, allow_redirects=True)
    print("Response for", url, response)

    soup = BeautifulSoup(response.content, 'html.parser')

    all_links = soup.select('table tbody tr div.fc-daygrid-day-events a')
    print(all_links)

result = get_website_news_links_thalheimCh()

I'm always getting [] for all_links variable. I guess that I'm doing something wrong. \Ihave looked on links that I found on 'Network' tab but I can not find any kind of link that can help me.

CodePudding user response：

All the data comes from a POST request.

You can get that with this:

import requests
from bs4 import BeautifulSoup

query_url = "https://thalheim.ch/index.php?option=com_dpcalendar&view=events&format=raw&limit=0&Itemid=504&ids=g-9&date-start=2021-11-29T00:00:00&date-end=2022-01-10T00:00:00"
page = requests.post(query_url).json()

for item in page["data"]:
    location = item["location"][0]["location"] if item["location"] else "N/A"
    desc = BeautifulSoup(item["description"], "lxml")
    print(f"Event: {item['title']} | Location: {location}")
    print(desc.getText(strip=True, separator=" "))
    print("-" * 120)

Output:

Event: Gemeindeversammlung | Location: Thalheim, Aargau 5112
09.12.2021 19:30 - 21:30 [Veranstaltungskalender Thalheim an der Thur] Gemeindeversammlung
------------------------------------------------------------------------------------------------------------------------
Event: Schulsilvester | Location: N/A
17.12.2021 [Veranstaltungskalender Thalheim an der Thur] Schulsilvester
------------------------------------------------------------------------------------------------------------------------
Event: Weihnachtsferien Schule | Location: N/A
20.12.2021 - 31.12.2021 [Veranstaltungskalender Thalheim an der Thur] Weihnachtsferien Schule
------------------------------------------------------------------------------------------------------------------------
Event: TV Thalheim - Abendunterhaltung 2022 (Frühvorstellung) | Location: 
29.12.2021 [Veranstaltungskalender Thalheim an der Thur] TV Thalheim - Abendunterhaltung 2022 (Frühvorstellung)
------------------------------------------------------------------------------------------------------------------------
Event: TV Thalheim - Abendunterhaltung 2022 | Location: 
31.12.2021 [Veranstaltungskalender Thalheim an der Thur] TV Thalheim - Abendunterhaltung 2022
------------------------------------------------------------------------------------------------------------------------
Event: TV Thalheim - Abendunterhaltung 2022 | Location: 
01.01.2022 [Veranstaltungskalender Thalheim an der Thur] TV Thalheim - Abendunterhaltung 2022
------------------------------------------------------------------------------------------------------------------------