Home > other >  Repeated timeouts when trying to scrape XML Data to a datatable (fatal SSL/TLS alert)
Repeated timeouts when trying to scrape XML Data to a datatable (fatal SSL/TLS alert)

Time:03-31

I'm trying to scrape xml data into a dataframe from this website:

enter image description here

However my code keeps throwing up errors all over the place: Such as: Error in open.connection(x, "rb") : Timeout was reached: [www.dmo.gov.uk] Connection timed out after 10001 milliseconds

Code is below:

library(data.table)
library(rvest)
library(xml2)

url <- read_html("https://www.dmo.gov.uk/data/XmlDataReport?reportCode=D1A")
dt <- rbindlist(lapply(url %>% html_nodes(css = "body > View_GILTS_IN_ISSUE > View_GILTS_IN_ISSUE") %>% 
                         xml_attrs(), 
                       function(x) as.data.table(t((x)))))
dt <- cbind(dt[,9, with = TRUE], 
            as.data.table(lapply(dt[,-9, with = TRUE], as.character)))
dt

Does anyone have any advice on how I can take this to completion?

CodePudding user response:

Was able to fix the issue with a combination of mkpt_uk's answer, and the one available here: enter image description here

  • Related