My objective with this code is to scrap the allocation of brazilian funds.
PATH = (r"C:\Users\gustavo.vieira\Desktop\python\drivers\msedgedriver.exe")
funds_list = ['VINLAND MACRO MASTER FUNDO DE INVESTIMENTO MULTIMERCADO']
url = 'https://cvmweb.cvm.gov.br/SWB/Sistemas/SCW/CPublica/CConsolFdo/FormBuscaParticFdo.aspx'
for i in funds_lista:
driver = webdriver.Edge(PATH)
driver.get(url)
search = driver.find_element_by_name("txtCNPJNome")
search.send_keys(i)
search.send_keys(Keys.RETURN)
driver.find_element_by_link_text(i).click()
font = driver.find_element_by_link_text('Composição da Carteira').click()
soup = BeautifulSoup(font)
rows = soup.find_all("tr")
print(rows)
My problem starts with bs4, I already tried to scrap the table using soup.find_all('table', id ='dlAplics'), or using (class_, all of them return an error. I already saw a lot of videos but I'm feeling unable to solve that by myself right now.
CodePudding user response：
What happens?
You assign the .click() to variable font and try to process it with BeautifulSoup what won't work.
How to fix?
Instead provide driver.page_source to BeautifulSoup to operate on the html.
Change:
font = driver.find_element_by_link_text('Composição da Carteira').click()
soup = BeautifulSoup(font)
to:
driver.find_element_by_link_text('Composição da Carteira').click()
soup = BeautifulSoup(driver.page_source)
Note * Id's in general are unique so you could use soup.find('table', id ='dlAplics') instead of find_all() to directly operate on this table.*
Output
<table border="1" cellpadding="4" cellspacing="0" id="dlAplics" rules="all" style="border-width:1px;border-style:solid;width:100%;border-collapse:collapse;">
<tbody><tr style="background-color:#ACC2D8;">
<td align="center" colspan="11"><b>Lista de Aplicações<br/>Clique sobre o ativo para mais informações.</b></td>
</tr><tr style="background-color:#ACC2D8;">
<td align="center" rowspan="3" style="width:30%;"><b>Ativo</b></td><td align="center" rowspan="3"><b>Classificação</b></td><td align="center" rowspan="3"><b>Empresa Ligada</b></td><td align="center" colspan="4"><b>Negócios Realizados no Mês</b></td><td align="center" colspan="4"><b>Posição Final</b></td>
</tr><tr style="background-color:#ACC2D8;">
<td align="center" colspan="2"><b>Vendas</b></td><td align="center" colspan="2"><b>Aquisições</b></td><td align="center" rowspan="2" style="width:7%;"><b>Quant.</b></td><td align="center" colspan="2"><b>Valores</b></td><td align="center" rowspan="2" style="width:7%;"><b>% Patr. Líq.</b></td>
</tr><tr style="background-color:#ACC2D8;">
<td align="center" style="width:7%;"><b>Quant.</b></td><td align="center" style="width:7%;"><b>Valor<br/></b></td><td align="center" style="width:7%;"><b>Quant.</b></td><td align="center" style="width:7%;"><b>Valor<br/></b></td><td align="center" style="width:7%;"><b>Custo<br/></b></td><td align="center" style="width:7%;"><b>Mercado<br/></b></td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375627')" style="cursor: hand; COLOR: olivedrab"><b></b> Ações<br/><b>Cod. Ativo:</b> RDOR3<br/><b>Dt. Ini. Vigen.:</b> 10/12/2020<br/></span></td><td align="center"><span id="dlAplics__ctl1_lbNConfTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl1_lbNConfEmpLigada">Não</span></td><td align="center">38.219</td><td align="center"><span id="dlAplics__ctl1_lbNConfVlVendasNegoc">2.139.131,81</span></td><td align="center">37.866</td><td align="center"><span id="dlAplics__ctl1_lbNConfVlAquisNegoc">2.239.218,96</span></td><td align="center">1.502</td><td align="center"><span id="dlAplics__ctl1_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl1_lbNConfVlPosFim">75.355,34</span></td><td align="center">0,005</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375615')" style="cursor: hand; COLOR: olivedrab"><b></b> Ações<br/><b>Cod. Ativo:</b> RDOR3<br/><b>Dt. Ini. Vigen.:</b> 25/06/2021<br/></span></td><td align="center"><span id="dlAplics__ctl2_lbNConfAltTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl2_lbNConfAltEmpLigada">Não</span></td><td align="center">38.219</td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlVendasNegoc">2.139.131,81</span></td><td align="center">37.866</td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlAquisNegoc">2.239.218,96</span></td><td align="center">1.502</td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl2_lbNConfAltVlPosFim">75.355,34</span></td><td align="center">0,005</td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375728')" style="cursor: hand; COLOR: olivedrab"><b></b> Certificado ou recibo de depósito de valores mobiliários<br/><b>Descrição:</b> TAEE11 - TRANS. ALIANCA DE ENERGIA ELETRICA S/A - TAEE11<br/><b>CNPJ do emissor:</b> 07.859.971/0001-30<br/><b>Denominação Social do emissor: </b> TRANS. ALIANCA DE ENERGIA ELETRICA S/A<br/></span></td><td align="center"><span id="dlAplics__ctl3_lbNConfTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl3_lbNConfEmpLigada">Não</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl3_lbNConfVlVendasNegoc">0,00</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl3_lbNConfVlAquisNegoc">0,00</span></td><td align="center">979</td><td align="center"><span id="dlAplics__ctl3_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl3_lbNConfVlPosFim">35.273,37</span></td><td align="center">0,002</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375488')" style="cursor: hand; COLOR: olivedrab"><b></b> Cotas de Fundos<br/><b></b> HASHDEX NASDAQ ETHEREUM REFERENCE PRICE FUNDO DE ÍNDICE<br/></span></td><td align="center"><span id="dlAplics__ctl4_lbNConfAltTpNegoc"></span></td><td align="center"><span id="dlAplics__ctl4_lbNConfAltEmpLigada">Não</span></td><td align="center">23.967</td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlVendasNegoc">1.667.612,78</span></td><td align="center">209.290</td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlAquisNegoc">15.823.222,00</span></td><td align="center">142.956</td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl4_lbNConfAltVlPosFim">11.436.480,00</span></td><td align="center">0,787</td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375486')" style="cursor: hand; COLOR: olivedrab"><b></b> Cotas de Fundos<br/><b></b> HASHDEX NASDAQ CRYPTO INDEX FUNDO DE INDICE<br/></span></td><td align="center"><span id="dlAplics__ctl5_lbNConfTpNegoc"></span></td><td align="center"><span id="dlAplics__ctl5_lbNConfEmpLigada">Não</span></td><td align="center">628.668</td><td align="center"><span id="dlAplics__ctl5_lbNConfVlVendasNegoc">38.177.765,73</span></td><td align="center">634.819</td><td align="center"><span id="dlAplics__ctl5_lbNConfVlAquisNegoc">39.878.659,26</span></td><td align="center">87.530</td><td align="center"><span id="dlAplics__ctl5_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl5_lbNConfVlPosFim">5.487.255,70</span></td><td align="center">0,378</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375484')" style="cursor: hand; COLOR: olivedrab"><b></b> Cotas de Fundos<br/><b></b> ISHARES S&P 500 FUNDO DE INVESTIMENTO EM COTAS DE FUNDO DE ÍNDICE - INVESTIMENTO NO EXTERIOR<br/></span></td><td align="center"><span id="dlAplics__ctl6_lbNConfAltTpNegoc"></span></td><td align="center"><span id="dlAplics__ctl6_lbNConfAltEmpLigada">Não</span></td><td align="center">26.057</td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlVendasNegoc">7.285.144,24</span></td><td align="center">31.466</td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlAquisNegoc">8.871.406,46</span></td><td align="center">5.983</td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl6_lbNConfAltVlPosFim">1.678.530,65</span></td><td align="center">0,116</td>
</tr><tr style="background-color:Cornsilk;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375640')" style="cursor: hand; COLOR: olivedrab"><b></b> Investimento no Exterior<br/><b>Nome do emissor: </b> CITCO - 376420,897566<br/><b>Venc.:</b> 31/12/2050<br/></span></td><td align="center"><span id="dlAplics__ctl7_lbNConfTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl7_lbNConfEmpLigada">Não</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl7_lbNConfVlVendasNegoc">0,00</span></td><td align="center">0</td><td align="center"><span id="dlAplics__ctl7_lbNConfVlAquisNegoc">0,00</span></td><td align="center">376.421</td><td align="center"><span id="dlAplics__ctl7_lbNConfVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl7_lbNConfVlPosFim">257.652.646,82</span></td><td align="center">17,731</td>
</tr><tr style="background-color:#FAEFCA;">
<td align="left"><span onclick="AbrirDet('CDADetAplic.aspx?PkCDAAplic=72375755')" style="cursor: hand; COLOR: olivedrab"><b></b> Valores a pagar<br/><b>Descrição:</b> AURA33<br/><b>CNPJ do emissor:</b> 07.857.093/0001-14<br/><b>Denominação Social do emissor: </b> AURA MINERALS INC.<br/></span></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltTpNegoc">Para<br/>negociação</span></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltEmpLigada"></span></td><td align="center"></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlVendasNegoc"></span></td><td align="center"></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlAquisNegoc"></span></td><td align="center"></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlCustoCorrecPosFim"></span></td><td align="center"><span id="dlAplics__ctl8_lbNConfAltVlPosFim">458.916,75</span></td><td align="center">-0,032</td>
</tr>
</tbody></table>CodePudding user response：
To scrape all the text from the first table on Consulta a Carteiras de Fundos page you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following Locator Strategies:
Code block:
driver.get("https://cvmweb.cvm.gov.br/SWB/Sistemas/SCW/CPublica/CConsolFdo/FormBuscaParticFdo.aspx") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#txtCNPJNome"))).send_keys("VINLAND MACRO MASTER FUNDO DE INVESTIMENTO MULTIMERCADO" Keys.RETURN) WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a#ddlFundos__ctl0_lnkbtn1"))).click() WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.PARTIAL_LINK_TEXT, "Composição da Carteira"))).click() print(WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//table[@id='tabAtivos']//table[@id='dlAplics']"))).text)Console output:
Lista de Aplicações Clique sobre o ativo para mais informações. Ativo Classificação Empresa Ligada Negócios Realizados no Mês Posição Final Vendas Aquisições Quant. Valores % Patr. Líq. Quant. Valor Quant. Valor Custo Mercado Ações Cod. Ativo: RDOR3 Dt. Ini. Vigen.: 10/12/2020 Para negociação Não 38.219 2.139.131,81 37.866 2.239.218,96 1.502 75.355,34 0,005 Ações Cod. Ativo: RDOR3 Dt. Ini. Vigen.: 25/06/2021 Para negociação Não 38.219 2.139.131,81 37.866 2.239.218,96 1.502 75.355,34 0,005 Certificado ou recibo de depósito de valores mobiliários Descrição: TAEE11 - TRANS. ALIANCA DE ENERGIA ELETRICA S/A - TAEE11 CNPJ do emissor: 07.859.971/0001-30 Denominação Social do emissor: TRANS. ALIANCA DE ENERGIA ELETRICA S/A Para negociação Não 0 0,00 0 0,00 979 35.273,37 0,002 Cotas de Fundos HASHDEX NASDAQ ETHEREUM REFERENCE PRICE FUNDO DE ÍNDICE Não 23.967 1.667.612,78 209.290 15.823.222,00 142.956 11.436.480,00 0,787 Cotas de Fundos HASHDEX NASDAQ CRYPTO INDEX FUNDO DE INDICE Não 628.668 38.177.765,73 634.819 39.878.659,26 87.530 5.487.255,70 0,378 Cotas de Fundos ISHARES S&P 500 FUNDO DE INVESTIMENTO EM COTAS DE FUNDO DE ÍNDICE - INVESTIMENTO NO EXTERIOR Não 26.057 7.285.144,24 31.466 8.871.406,46 5.983 1.678.530,65 0,116 Investimento no Exterior Nome do emissor: CITCO - 376420,897566 Venc.: 31/12/2050 Para negociação Não 0 0,00 0 0,00 376.421 257.652.646,82 17,731 Valores a pagar Descrição: AURA33 CNPJ do emissor: 07.857.093/0001-14 Denominação Social do emissor: AURA MINERALS INC. Para negociação 458.916,75 -0,032