finding alternate versions of a tag when iteratively scraping many links with rvest-CodePudding

I am scraping some data from the sec archives. Each xml document has the basic form:

<ns1:infoTable>
<ns1:nameOfIssuer>ACCENTURE PLC IRELAND</ns1:nameOfIssuer>
<ns1:titleOfClass>SHS CLASS A</ns1:titleOfClass>
<ns1:cusip>G1151C101</ns1:cusip>
<ns1:value>47837</ns1:value>
<ns1:shrsOrPrnAmt>
<ns1:sshPrnamt>183135</ns1:sshPrnamt>
<ns1:sshPrnamtType>SH</ns1:sshPrnamtType>
</ns1:shrsOrPrnAmt>
<ns1:investmentDiscretion>SOLE</ns1:investmentDiscretion>
<ns1:votingAuthority>
<ns1:Sole>0</ns1:Sole>
<ns1:Shared>0</ns1:Shared>
<ns1:None>183135</ns1:None>
</ns1:votingAuthority>
</ns1:infoTable>

However, occasionally documents have the form:

<infoTable>
<nameOfIssuer>2U INC</nameOfIssuer>
<titleOfClass>COM</titleOfClass>
<cusip>90214J101</cusip>
<value>340</value>
<shrsOrPrnAmt>
<sshPrnamt>8504</sshPrnamt>
<sshPrnamtType>SH</sshPrnamtType>
</shrsOrPrnAmt>
<investmentDiscretion>SOLE</investmentDiscretion>
<votingAuthority>
<Sole>8504</Sole>
<Shared>0</Shared>
<None>0</None>
</votingAuthority>
</infoTable>

So the only difference in the tags is the addition of the "ns1:" prefix.

While scraping the data I am able to find nodes like so:

urll <- "https://www.sec.gov/Archives/edgar/data/1002152/000108514621000479/infotable.xml"

session %>% 
    nod(urll) %>% 
    scrape(verbose = FALSE) %>%
    xml_ns_strip() %>% 
    xml_find_all('ns1:infoTable')

or for the alternate tag that doesn't have the ns1: prefix

urll <- "https://www.sec.gov/Archives/edgar/data/1002672/000106299321000915/form13fInfoTable.xml"

session %>% 
  nod(urll) %>% 
  scrape(verbose = FALSE) %>%
  xml_ns_strip() %>% 
  xml_find_all('infoTable')

But when looping over a number of links I do not know apriori which xml document will have which tag. Is there a way to get the nodes either by specifying the nodes with an "or" operator or finding the tag with a string match looking for the specific text "infoTable" in the tag?

I tried:

session %>% 
  nod(urll) %>% 
  scrape(verbose = FALSE) %>%
  xml_ns_strip() %>% 
  xml_find_all(xpath = '//*[self::infoTable or self::ns1:infoTable]')

session %>% 
  nod(urll) %>% 
  scrape(verbose = FALSE) %>%
  xml_ns_strip() %>% 
  xml_find_all(xpath = "//*[contains(text(),'infoTable')]")

But neither variation works. Any suggestions on how to get it to work?

Thanks in advance. I am using polite, rvest, dplyr

CodePudding user response：

Consider local-name() in your XPath expression. Below uses httr and the new R 4.1.0 pipe |>:

library(xml2)
library(httr)

url <- "https://www.sec.gov/Archives/edgar/data/1002152/000108514621000479/infotable.xml"

info_tables <- httr::GET(url, user_agent("Mozilla/5.0")) |> 
  httr::content(encoding="UTF-8") |>
  xml2::xml_find_all(xpath = "//*[local-name()='infoTable']")

And to build dataframe:

df_list <- lapply(info_tables, function(r) {
  vals <- xml2::xml_children(r)
  
  other_vals <- xml2::xml_find_all(r, "*") |>
    xml2::xml_children()

  child_df <- setNames(
    c(xml2::xml_text(vals)), 
    c(xml2::xml_name(vals))
  ) |> rbind() |> data.frame()
  
  grand_df <- setNames(
    c(xml2::xml_text(other_vals)), 
    c(xml2::xml_name(other_vals))
  ) |> rbind() |> data.frame()
  
  cbind.data.frame(child_df, grand_df)
})

final_df <- do.call(rbind.data.frame, df_list)
final_df
                   nameOfIssuer titleOfClass     cusip value shrsOrPrnAmt investmentDiscretion votingAuthority sshPrnamt sshPrnamtType Sole Shared   None
1         ACCENTURE PLC IRELAND  SHS CLASS A G1151C101 47837     183135SH                 SOLE        00183135    183135            SH    0      0 183135
2                  ALPHABET INC CAP STK CL A 02079K305 43695      24931SH                 SOLE         0024931     24931            SH    0      0  24931
3                     APPLE INC          COM 037833100  3229      24334SH                 SOLE         0024334     24334            SH    0      0  24334
4    BERKSHIRE HATHAWAY INC DEL         CL A 084670108  2783          8SH                 SOLE             008         8            SH    0      0      8
5           CANADIAN NATL RY CO          COM 136375102   218       1985SH                 SOLE          001985      1985            SH    0      0   1985
6  CHECK POINT SOFTWARE TECH LT          ORD M22465104 45505     342375SH                 SOLE        00342375    342375            SH    0      0 342375
7           CHURCH & DWIGHT INC          COM 171340102 42500     487221SH                 SOLE        00487221    487221            SH    0      0 487221
8  COGNIZANT TECHNOLOGY SOLUTIO         CL A 192446102 46076     562243SH                 SOLE        00562243    562243            SH    0      0 562243
9               CVS HEALTH CORP          COM 126650100 44311     648773SH                 SOLE        00648773    648773            SH    0      0 648773
10          DANAHER CORPORATION          COM 235851102 44200     198974SH                 SOLE        00198974    198974            SH    0      0 198974