I am trying to extract all the headlines from this website: https://www.theguardian.com/international. I have tried so many xpaths, but none gives me the headlines, although the code works. Any ideas on what I'm doing wrong? Thanks!
This is my code:
guardian_url <- "https://www.theguardian.com/international"
guardian <- read_html(guardian_url)
headlines <- guardian %>%
html_elements(xpath = '//html/body/div[3]/div') %>%
html_text2()
CodePudding user response：
library(tidyverse)
library(rvest)
#>
#> Attaching package: 'rvest'
#> The following object is masked from 'package:readr':
#>
#> guess_encoding
"https://www.theguardian.com/international" %>%
read_html %>%
html_elements(".fc-item__title") %>%
html_text2()
#> [1] "UK Dover asylum centre not operating legally, concedes minister"
#> [2] "Exclusive Bahraini death row prisoner pleads with pope to aid his release"
#> [3] "Live Russia-Ukraine war: Blinken hails Turkish help as grain export deal resumes"
#> [4] "‘We’re facing a defining moment’ Biden urges Americans to take a stand against political violence"
#> [5] "Live UK: biggest interest rate rise in decades expected as Bank of England battles inflation"
#> [6] "North Korea ICBM launch may have been a failure, South’s military says"
#> [7] "Twitter Company may ‘halve its workforce’ as key investor backs job cuts"
#> [8] "China Father of three-year-old blames zero-Covid policy for son’s death"
#> [9] "Hong Kong Exiles in UK unnerved by ‘weak’ response to beating of protester"
#> [10] "Myanmar Experts warn of aid ‘catastrophe’, after junta law change"
#> [11] "Analysis Grain deal U-turn offers lesson in calling Putin’s bluff"
#> [12] "At a glance What we know on day 253 of the invasion"
#> [13] "UK Kremlin to summon British ambassador over drone attacks on Black Sea fleet"
#> [14] "Roman Abramovich UK adds two ‘business associates’ of former Chelsea owner to Russia sanctions list"
#> [15] "‘He is poised to open the floodgates’ Can Twitter survive Elon Musk – or even thrive?"
#> [16] "The long read The many meanings of moss"
#> [17] "No fun rides but plenty of spirit Studio Ghibli offers anime fans a new walk in the park"
#> [18] "Germany's balancing act Scholz heads to China amid questions over strategy"
#> [19] "Weird: The Al Yankovic Story review Daniel Radcliffe biopic packed with wacky walk-ons"
#> [20] "‘No darkness is for ever’ Can an activist in exile persuade the Taliban to allow teaching on TV?"
#> [21] "Nil By Mouth review Gary Oldman’s overwhelming study of family violence"
#> [22] "‘More dignity’ Aid organisations switch to cash in drought-hit Ethiopia"
#> [23] "Europe Climate warming at twice rate of global average, says report"
#> [24] "No more drinking water, little food: our island is a field of bones"
#> [25] "Rishi Sunak PM U-turns on decision not to attend Cop27"
#> [26] "Cop27 Egyptian hunger striker may die in prison, Nobel laureates warn world leaders attending summit"
#> [27] "Fossil fuel burning once caused a mass extinction – now we’re risking another"
#> [28] "World leaders at Cop27 in Egypt must demand the release of Alaa Abd El-Fattah"
#> [29] "The ProPublica-Vanity Fair report on Covid-19’s origins is explosive. Is it reliable?"
#> [30] "Benjamin Netanyahu may be back – but the true victory belongs to Israel’s far right"
#> [31] "Can you ‘lose’ an accent? And more importantly, why would you want to?"
#> [32] "With the global economy on the rocks, quiet quitting is no longer a thing"
#> [33] "Qatar calling its critics racist opens a debate that may be worth having"
#> [34] "Pancakes, shakes and KFC When Barkley tried to eat his way out of the 76ers"
#> [35] "‘Any problem you have, it’s nothing’ Rico’s brotherly inspiration"
#> [36] "World Series Astros silence Phillies in Game 4 with rare no-hitter"
#> [37] "Cricket Yorkshire racism hearing to be held in public after request from Rafiq"
#> [38] "T20 World Cup Australia wary of ‘pushing too hard’ to improve run rate"
#> [39] "Chelsea 2-1 Dinamo Zagreb Chilwell set to miss World Cup with hamstring injury"
#> [40] "To criticise or not to criticise Are women’s football pundits too nice?"
#> [41] "Europe French-German friendship ‘still alive’ as Macron meets Scholz amid tensions"
#> [42] "‘Nobody forced us’ The Greek builder who saved 80 Afghans from the sea"
#> [43] "Italy Fall of Liz Truss gives far right a lesson in what not to do"
#> [44] "‘Get Igor Girkin’ Hopes MH17 suspect could be captured fighting in Ukraine"
#> [45] "Israel election Netanyahu thanks voters as rightwing bloc extends Israeli election lead"
#> [46] "Africa Invasive mosquito could disrupt ‘landscape of malaria’ after cases rise"
#> [47] "Twitter exodus Company faces murky future as top managers flee the nest"
#> [48] "Climate crisis Big agriculture warns farming must change or risk ‘destroying the planet’"
#> [49] "Hillsong Church Founder Brian Houston breaks silence with video stating ‘I will fight’ criminal charge in Australia"
#> [50] "Denmark election Result keeps Social Democrats at the helm"
#> [51] "Seoul crowd crush Local police offices raided in investigation"
#> [52] "Exclusive UN poverty envoy tells Britain this is ‘worst time’ for more austerity"
#> [53] "Ethiopian civil war Parties agree truce to end hostilities"
#> [54] "Just Stop Oil Protesters who targeted Girl with a Pearl Earring jailed by Dutch court"
#> [55] "Good Night Oppy review Cutesy Spielberg-assisted Mars documentary"
#> [56] "Dinosaurs by Lydia Millet review Can a wealthy man be good?"
#> [57] "‘Did I notice a dark side?’ The true story behind serial killer drama The Good Nurse"
#> [58] "Selena Gomez My Mind & Me review: a fascinating and frustrating pop doc"
#> [59] "Avatar: The Way of Water James Cameron releases extended trailer"
#> [60] "Killer Sally review A warped true-crime tale of bodybuilder murder"
#> [61] "How to cook the perfect ... Pumpkin gnocchi"
#> [62] "Rail route of the month The slow train to Skagen, Denmark, where the North Sea meets the Baltic"
#> [63] "A moment that changed me I threw away the sheet music and all the colour and passion I had for the harp came back to me"
#> [64] "Pushing Buttons Pushing Buttons: Freaky games form some of my most vivid childhood memories"
#> [65] "Kitchen aide Why parmesan rinds are a cook’s secret ingredient"
#> [66] "From pirates’ hideout to Dalí’s bolthole Cadaqués, star of Spain’s Costa Brava"
#> [67] "US midterm elections 2022 Republicans and Democrats spend big on ads"
#> [68] "US midterm elections 2022 Why is Nike founder Phil Knight so desperate to prevent a Democratic win in Oregon?"
#> [69] "Cambodia’s modern slavery nightmare The human trafficking crisis overlooked by authorities"
#> [70] "Psychology I was an unhappy teenager, among lonely people, in thrall to a charismatic leader – had I joined a cult?"
#> [71] "‘This is apartheid’ Cape Town slum residents condemn forced removals"
#> [72] "‘I can’t work in the office safely’ The over-50s leaving the UK labour force"
#> [73] "Russians Tell us what you think about Putin’s escalation of war in Ukraine"
#> [74] "Iranians Share your views on the protests following Mahsa Amini’s death"
#> [75] "Tell us How have you been affected by the situation in Ukraine?"
#> [76] "Get in touch Share a story with the Guardian"
#> [77] "Podcast Can Twitter survive Elon Musk? (And can Musk survive Twitter?)"
#> [78] "Listen to previous episodes"
#> [79] "Anywhere but Washington Moral panic, culture wars and Ron DeSantis: will Florida stay red in 2022?"
#> [80] "Ukraine A village brutalised by Russia, and the youth rebuilding homes and hope"
#> [81] "Rishi Sunak How he became UK PM: three days in three minutes"
#> [82] "China Could Xi follow Putin's example and try to annex Taiwan?"
#> [83] "US Can abortion rights swing the midterm elections?"
#> [84] "Heating and eating Can cost of living and climate protesters join forces?"
#> [85] "Ukraine How Russia’s strategy failed, not the tank"
#> [86] "'I feel like the Beyoncé of Brazilian politics' The trans woman fighting back against Bolsonaro"
#> [87] "Iran What the latest footage tells us about the regime"
#> [88] "Galloping horses, spears and adrenaline In tent pegging 'anything can happen'"
#> [89] "‘Extraordinary treasure trove’ Saul Leiter’s unseen images"
#> [90] "Wednesday’s best photos Panda diplomacy and circus skills"
#> [91] "The things left behind Seould police display belongings found after crowd crush – in pictures"
#> [92] "Indigenous Australians Vigils held across Australia for Cassius Turvey"
#> [93] "Village people The rituals of rural life"
#> [94] "Mexico Day of the Dead celebrations"
#> [95] "Grain deal U-turn offers lesson in calling Vladimir Putin’s bluff"
#> [96] "Live Russia-Ukraine war live news: Blinken hails Turkish help as grain export deal resumes"
#> [97] "Hong Kong exiles in UK unnerved by ‘weak’ response to beating of protester"
#> [98] "‘We need your help’: young girl throws note over fence at Manston"
#> [99] "North Korea ICBM launch may have been a failure, South’s military says"
#> [100] "Zelenskiy labels Putin U-turn on Ukraine grain deal a ‘failure of Russian aggression’"
#> [101] "Bahraini death row prisoner pleads with pope to aid his release"
#> [102] "No more drinking water, little food: our island is a field of bones"
#> [103] "Russia-Ukraine war at a glance: what we know on day 253 of the invasion"
#> [104] "Father of three-year-old blames China’s zero-Covid policy for son’s death"
Created on 2022-11-03 with reprex v2.0.2
CodePudding user response：
//a[@data-link-name='article'and not(contains(@class, 'u-faux-block-link__overlay'))]
This picks the headlines and the subheadlines as well.