bs4 select_one on loop sometimes has lack of a value text and fails how should i fix it?-CodePudding

    url=http://www.mercadopublico.cl/Procurement/Modules/RFB/DetailsAcquisition.aspx?idlicitacion=3951-24-L122
    print(i)
    soup = BeautifulSoup(requests.get(i).content, "html.parser")
    soup.select("#box_a2 .box_a_text_00")
    licitation_number = soup.select_one("#lblNumLicitacion").text
    responsable = soup.select_one("#lblResponsable").text
    ficha = soup.select_one("#lblFicha2Reclamo").text
    nombre_licitacion=soup.select_one("#lblNombreLicitacion").text
    
    
    try:
        estado_licitacion = soup.select_one("#lblFicha1Estado").text
    except ValueError:

i have severals links where I make my loop

sometimas this part

estado_licitacion = soup.select_one("#lblFicha1Estado").text

get erros because does not exist, tryed to fixed with try and except but keep getting this error

AttributeError: 'NoneType' object has no attribute 'text'

when doesnt exist, the example is done for error, the loop doesnt fail all the time

CodePudding user response：

if soup.select_one("#lblFicha1Estado").text:
    estado_licitacion = soup.select_one("#lblFicha1Estado").text
else:
    pass

CodePudding user response：

You should check if your element is available in your soup:

estado_licitacion = e.text if (e:=soup.select_one("#lblFicha1Estado")) else 'no estato licitation available'

Note walrus operator requires Python 3.8 or later to work.*

Alternative for < Python 3.8:

estado_licitacion = soup.select_one("#lblFicha1Estado").text if soup.select_one("#lblFicha1Estado") else 'no estato licitation available'

Example

import requests
from bs4 import BeautifulSoup

idList = ['3951-24-L122']
data = []
for idlicitacion in idList:
    url=f'http://www.mercadopublico.cl/Procurement/Modules/RFB/DetailsAcquisition.aspx?idlicitacion={idlicitacion}'
    soup = BeautifulSoup(requests.get(url).content)

    data.append({
        'licitation_number': e.text if (e:=soup.select_one("#lblNumLicitacion")) else 'no licitation number available',
        'responsable': e.text if (e:=soup.select_one("#lblResponsable")) else 'no responsable available',
        'ficha': e.text if (e:=soup.select_one("#lblFicha2Reclamo")) else 'no ficha available',
        'estado_licitacion': e.text if (e:=soup.select_one("#lblFicha1Estado")) else 'no estato licitation available'
    })

data

Output

[{'licitation_number': '3951-24-L122', 'responsable': 'CONSULTORIO GENERAL URBANO VICTOR DOMINGO SILVA, Consultorio Victor Domingo Silva', 'ficha': '0', 'estado_licitacion': 'no estato licitation available'}]