I'm attempting to get data from Wikipedias sidebar on the 'Current Events' page with the below. At the moment this produces an array of Objects each with value title and url.

I would also like to provide a new value to the objects in array headline derived from the <h3> id or text content. This would result in each object having three values: headline, url and title. However, I'm unsure how to iterate through these.

Beautiful Soup Code

        soup = BeautifulSoup(response, "html.parser").find('div', {'aria-labelledby': 'Ongoing_events'})
        links = soup.findAll('a')


        for item in links:
            title = item.text
            url = ("https://en.wikipedia.org" item['href'])

            eo = CurrentEventsObject(title, url)
            eventsArray.append(eo)

Wikipedia Current Events List

<div class="mw-collapsible-content">
    <h3><span class="mw-headline" id="Disasters">Disasters</span</h3>
    <ul>
        <li><a href="/wiki/Climate_crisis" title="Climate crisis">Climate crisis</a></li>
        <li><a href="/wiki/COVID-19_pandemic" title="COVID-19 pandemic">COVID-19 pandemic</a></li>
        <li><a href="/wiki/2021–22_European_windstorm_season" title="2021–22 European windstorm season">2021–22 European windstorm season</a></li>
        <li><a href="/wiki/2020–21_H5N8_outbreak" class="mw-redirect" title="2020–21 H5N8 outbreak">2020–21 H5N8 outbreak</a></li>
        <li><a href="/wiki/2021_Pacific_typhoon_season" title="2021 Pacific typhoon season">2021 Pacific typhoon season</a></li>
        <li><a href="/wiki/2021_Madagascar_food_crisis" class="mw-redirect" title="2021 Madagascar food crisis">Madagascar food crisis</a></li>
        <li><a href="/wiki/Water_scarcity_in_Iran" title="Water scarcity in Iran">Water crisis in Iran</a></li>
        <li><a href="/wiki/Famine_in_Yemen_(2016–present)" title="Famine in Yemen (2016–present)">Yemeni famine</a></li>
        <li><a href="/wiki/2021_Cumbre_Vieja_volcanic_eruption" title="2021 Cumbre Vieja volcanic eruption">2021 La Palma eruption</a></li>
    </ul>
    <h3><span class="mw-headline" id="Economic">Economic</span></h3>
    <ul>
        <li><a href="/wiki/2020–2021_global_chip_shortage" title="2020–2021 global chip shortage">2020–2021 global chip shortage</a></li>
        <li><a href="/wiki/2021_global_supply_chain_crisis" title="2021 global supply chain crisis">2021 global supply chain crisis</a></li>
        <li><a href="/wiki/COVID-19_recession" title="COVID-19 recession">COVID-19 recession</a></li>
        <li><a href="/wiki/Lebanese_liquidity_crisis" title="Lebanese liquidity crisis">Lebanese liquidity crisis</a></li>
        <li><a href="/wiki/Pandora_Papers" title="Pandora Papers">Pandora Papers leak</a></li>
        <li><a href="/wiki/2021_Sri_Lankan_economic_crisis" title="2021 Sri Lankan economic crisis">Sri Lankan economic and food crisis</a></li>
        <li><a href="/wiki/2018–2021_Turkish_currency_and_debt_crisis" title="2018–2021 Turkish currency and debt crisis">Turkish currency and debt crisis</a></li>
        <li><a href="/wiki/2021_United_Kingdom_natural_gas_supplier_crisis" title="2021 United Kingdom natural gas supplier crisis">United Kingdom natural gas supplier crisis</a></li>
    </ul>
    <h3><span class="mw-headline" id="Politics">Politics</span></h3>
    <ul>
        <li><a href="/wiki/2021_Belarus–European_Union_border_crisis" title="2021 Belarus–European Union border crisis">Belarus−European Union border crisis</a></li>
        <li><a href="/wiki/2021_Brazilian_protests" title="2021 Brazilian protests">Brazilian protests</a></li>
        <li><a href="/wiki/2021_Colombian_protests" title="2021 Colombian protests">Colombian tax reform protests</a></li>
        <li><a href="/wiki/2021_Eswatini_protests" title="2021 Eswatini protests">Eswatini protests</a></li>
        <li><a href="/wiki/2018–2021_Haitian_protests" title="2018–2021 Haitian protests">Haitian protests</a></li>
        <li><a href="/wiki/2020–2021_Indian_farmers'_protest" title="2020–2021 Indian farmers' protest">Indian farmers' protests</a></li>
        <li><a href="/wiki/Insulate_Britain_protests" title="Insulate Britain protests">Insulate Britain protests</a></li>
        <li><a href="/wiki/2021_Jersey_dispute" title="2021 Jersey dispute">Jersey dispute</a></li>
        <li><a href="/wiki/Libyan_peace_process" title="Libyan peace process">Libyan peace process</a></li>
        <li><a href="/wiki/2020–21_Malaysian_political_crisis" title="2020–21 Malaysian political crisis">Malaysian political crisis</a></li>
        <li><a href="/wiki/2021_Myanmar_protests" title="2021 Myanmar protests">Myanmar protests</a></li>
        <li><a href="/wiki/2018–2021_Nicaraguan_protests" title="2018–2021 Nicaraguan protests">Nicaraguan protests</a></li>
        <li><a href="/wiki/End_SARS" title="End SARS">Nigerian protests</a></li>
        <li><a href="/wiki/2019–2021_Persian_Gulf_crisis" title="2019–2021 Persian Gulf crisis">Persian Gulf crisis</a></li>
        <li><a href="/wiki/2017–present_Peruvian_political_crisis" title="2017–present Peruvian political crisis">Peruvian crisis</a></li>
        <li><a href="/wiki/2021_Russian_election_protests" title="2021 Russian election protests">Russian election protests</a></li>
        <li><a href="/wiki/2021_Solomon_Islands_unrest" title="2021 Solomon Islands unrest">Solomon Islands unrest</a></li>
        <li><a href="/wiki/Tigrayan_peace_process" title="Tigrayan peace process">Tigrayan peace process</a></li>
        <li><a href="/wiki/2020–2021_Thai_protests" title="2020–2021 Thai protests">Thai protests</a></li>
        <li><a href="/wiki/2021_Tunisian_political_crisis" title="2021 Tunisian political crisis">Tunisian political crisis</a></li>
        <li><a href="/wiki/2020–2021_United_States_racial_unrest" title="2020–2021 United States racial unrest">United States racial unrest</a></li>
        <li><a href="/wiki/Venezuelan_presidential_crisis" title="Venezuelan presidential crisis">Venezuelan presidential crisis</a></li>
    </ul>
    <div class="editlink noprint plainlinks"><a class="external text" href="https://en.wikipedia.org/w/index.php?title=Portal:Current_events/Sidebar&amp;action=edit">edit section</a></div>
</div>

CodePudding user response：

Note: Try to select your elements more specific to get all information in one process - Defining a list outside your loops will avoid from overwriting

Following steps will create a list of dicts, that for example could simply iterated or turned into a data frame.

#1 Select all <ul> that are direct siblings of a <h3>

soup.select('h3   ul')

#2 Select the <h3> and get its text:

e.find_previous_sibling('h3').text.strip()

#3 Select all <a> in the <ul> and iterat the results while creating a list of dicts:

for a in e.select('a'):
    data.append({
        'headline':headline,
        'title': a['title'],
        'url':'https://en.wikipedia.org' a['href']
})

Example

soup = BeautifulSoup(response, "html.parser").find('div', {'aria-labelledby': 'Ongoing_events'})

data = []

for e in soup.select('h3   ul'):
    headline = e.find_previous_sibling('h3').text.strip()
    for a in e.select('a'):
        data.append({
            'headline':headline,
            'title': a['title'],
            'url':'https://en.wikipedia.org' a['href']
        })

data

Output

[{'headline': 'Disasters',
  'title': 'Climate crisis',
  'url': 'https://en.wikipedia.org/wiki/Climate_crisis'},
 {'headline': 'Disasters',
  'title': 'COVID-19 pandemic',
  'url': 'https://en.wikipedia.org/wiki/COVID-19_pandemic'},
 {'headline': 'Disasters',
  'title': '2021–22 European windstorm season',
  'url': 'https://en.wikipedia.org/wiki/2021–22_European_windstorm_season'},
 {'headline': 'Disasters',
  'title': '2020–21 H5N8 outbreak',
  'url': 'https://en.wikipedia.org/wiki/2020–21_H5N8_outbreak'},
 {'headline': 'Disasters',
  'title': '2021 Pacific typhoon season',
  'url': 'https://en.wikipedia.org/wiki/2021_Pacific_typhoon_season'},
 {'headline': 'Disasters',
  'title': '2021 Madagascar food crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Madagascar_food_crisis'},
 {'headline': 'Disasters',
  'title': 'Water scarcity in Iran',
  'url': 'https://en.wikipedia.org/wiki/Water_scarcity_in_Iran'},
 {'headline': 'Disasters',
  'title': 'Famine in Yemen (2016–present)',
  'url': 'https://en.wikipedia.org/wiki/Famine_in_Yemen_(2016–present)'},
 {'headline': 'Disasters',
  'title': '2021 Cumbre Vieja volcanic eruption',
  'url': 'https://en.wikipedia.org/wiki/2021_Cumbre_Vieja_volcanic_eruption'},
 {'headline': 'Economic',
  'title': '2020–2021 global chip shortage',
  'url': 'https://en.wikipedia.org/wiki/2020–2021_global_chip_shortage'},
 {'headline': 'Economic',
  'title': '2021 global supply chain crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_global_supply_chain_crisis'},
 {'headline': 'Economic',
  'title': 'COVID-19 recession',
  'url': 'https://en.wikipedia.org/wiki/COVID-19_recession'},
 {'headline': 'Economic',
  'title': 'Lebanese liquidity crisis',
  'url': 'https://en.wikipedia.org/wiki/Lebanese_liquidity_crisis'},
 {'headline': 'Economic',
  'title': 'Pandora Papers',
  'url': 'https://en.wikipedia.org/wiki/Pandora_Papers'},
 {'headline': 'Economic',
  'title': '2021 Sri Lankan economic crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Sri_Lankan_economic_crisis'},
 {'headline': 'Economic',
  'title': '2018–2021 Turkish currency and debt crisis',
  'url': 'https://en.wikipedia.org/wiki/2018–2021_Turkish_currency_and_debt_crisis'},
 {'headline': 'Economic',
  'title': '2021 United Kingdom natural gas supplier crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_United_Kingdom_natural_gas_supplier_crisis'},
 {'headline': 'Politics',
  'title': '2021 Belarus–European Union border crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Belarus–European_Union_border_crisis'},
 {'headline': 'Politics',
  'title': '2021 Brazilian protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Brazilian_protests'},
 {'headline': 'Politics',
  'title': '2021 Colombian protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Colombian_protests'},
 {'headline': 'Politics',
  'title': '2021 Eswatini protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Eswatini_protests'},
 {'headline': 'Politics',
  'title': '2018–2021 Haitian protests',
  'url': 'https://en.wikipedia.org/wiki/2018–2021_Haitian_protests'},
 {'headline': 'Politics',
  'title': "2020–2021 Indian farmers' protest",
  'url': 'https://en.wikipedia.org/wiki/2020–2021_Indian_farmers'_protest'},
 {'headline': 'Politics',
  'title': 'Insulate Britain protests',
  'url': 'https://en.wikipedia.org/wiki/Insulate_Britain_protests'},
 {'headline': 'Politics',
  'title': '2021 Jersey dispute',
  'url': 'https://en.wikipedia.org/wiki/2021_Jersey_dispute'},
 {'headline': 'Politics',
  'title': 'Libyan peace process',
  'url': 'https://en.wikipedia.org/wiki/Libyan_peace_process'},
 {'headline': 'Politics',
  'title': '2020–21 Malaysian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2020–21_Malaysian_political_crisis'},
 {'headline': 'Politics',
  'title': '2021 Myanmar protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Myanmar_protests'},
 {'headline': 'Politics',
  'title': '2018–2021 Nicaraguan protests',
  'url': 'https://en.wikipedia.org/wiki/2018–2021_Nicaraguan_protests'},
 {'headline': 'Politics',
  'title': 'End SARS',
  'url': 'https://en.wikipedia.org/wiki/End_SARS'},
 {'headline': 'Politics',
  'title': '2019–2021 Persian Gulf crisis',
  'url': 'https://en.wikipedia.org/wiki/2019–2021_Persian_Gulf_crisis'},
 {'headline': 'Politics',
  'title': '2017–present Peruvian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2017–present_Peruvian_political_crisis'},
 {'headline': 'Politics',
  'title': '2021 Russian election protests',
  'url': 'https://en.wikipedia.org/wiki/2021_Russian_election_protests'},
 {'headline': 'Politics',
  'title': '2021 Solomon Islands unrest',
  'url': 'https://en.wikipedia.org/wiki/2021_Solomon_Islands_unrest'},
 {'headline': 'Politics',
  'title': 'Tigrayan peace process',
  'url': 'https://en.wikipedia.org/wiki/Tigrayan_peace_process'},
 {'headline': 'Politics',
  'title': '2020–2021 Thai protests',
  'url': 'https://en.wikipedia.org/wiki/2020–2021_Thai_protests'},
 {'headline': 'Politics',
  'title': '2021 Tunisian political crisis',
  'url': 'https://en.wikipedia.org/wiki/2021_Tunisian_political_crisis'},
 {'headline': 'Politics',
  'title': '2020–2021 United States racial unrest',
  'url': 'https://en.wikipedia.org/wiki/2020–2021_United_States_racial_unrest'},
 {'headline': 'Politics',
  'title': 'Venezuelan presidential crisis',
  'url': 'https://en.wikipedia.org/wiki/Venezuelan_presidential_crisis'}]