I have been following this tutorial: https://kb.objectrocket.com/postgresql/scrape-a-website-to-postgres-with-python-938

My app.py file looks like this (taken from the above tutorial):

from flask import Flask  # needed for flask-dependent libraries below
from flask import render_template  # to render the error page
from selenium import webdriver  # to grab source from URL
from bs4 import BeautifulSoup  # for searching through HTML
import psycopg2  # for database access

# set up Postgres database connection and cursor.
t_host = "localhost" # either "localhost", a domain name, or an IP address.
t_port = "5432" # default postgres port
t_dbname = "scrape"
t_user = "postgres"
t_pw = "********"
db_conn = psycopg2.connect(host=t_host, port=t_port, dbname=t_dbname, user=t_user, password=t_pw)
db_cursor = db_conn.cursor()

app = Flask(__name__)


@app.route("/")
@app.route('/import_temp')
def import_temp():
    # set up your webdriver to use Chrome web browser
    my_web_driver = webdriver.Chrome("/usr/local/bin/chromedriver")

    # designate the URL we want to scrape
    #   NOTE: the long string of characters at the end of this URL below is a clue that
    #   maybe this page is so dynamic, like maybe refers to a specific web session and/or day/time,
    #   that we can't necessarily count on it to be the same more than one time.
    #   Which means... we may want to find another source for our data; one that is more
    #   dependable. That said, whatever URL you use, the methodology in this lesson stands.
    t_url = "https://weather.com/weather/today/l/7ebb344012f0c5ff88820d763da89ed94306a86c770fda50c983bf01a0f55c0d"
    # initiate scrape of website page data
    my_web_driver.get(t_url)
    # return entire page into "t_content"
    t_content = my_web_driver.page_source
    # use soup to make page content easily searchable
    soup_in_bowl = BeautifulSoup(t_content, 'html.parser')
    # search for the UNIQUE span and class for the data we are looking for:
    o_temp = soup_in_bowl.find('span', attrs={'class': 'deg-feels'})
    # from the resulting object, "o_temp", get the text parameter and assign it to "n_temp"
    n_temp = o_temp.text

    # Build SQL for purpose of:
    #    saving the temperature data to a new row
    s = ""
    s  = "INSERT INTO tbl_temperatures"
    s  = "("
    s  = "n_temp"
    s  = ") VALUES ("
    s  = "(%n_temp)"
    s  = ")"

    # Trap errors for opening the file
    try:
        db_cursor.execute(s, [n_temp, n_temp])
        db_conn.commit()
    except psycopg2.Error as e:
        t_msg = "Database error: "   e   "/n open() SQL: "   s
        return render_template("error_page.html", t_msg = t_msg)

    # Success!
    # Show a message to user.
    t_msg = "Successful scrape!"
    return render_template("progress.html", t_msg = t_msg)

    # Clean up the cursor and connection objects
    db_cursor.close()
    db_conn.close()

I have successfully imported the required libraries, and created a database to store the scraped data. However, when I run the application I now receive an error:

FLASK_APP = app.py
FLASK_ENV = development
FLASK_DEBUG = 0
In folder /home/lloyd/PycharmProjects/flaskProject
/home/lloyd/PycharmProjects/flaskProject/venv/bin/python -m flask run
 * Serving Flask app 'app.py' (lazy loading)
 * Environment: development
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL C to quit)
[2021-12-29 20:51:32,817] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/lloyd/PycharmProjects/flaskProject/app.py", line 41, in import_temp
    n_temp = o_temp.text
AttributeError: 'NoneType' object has no attribute 'text'
127.0.0.1 - - [29/Dec/2021 20:51:32] "GET / HTTP/1.1" 500 -

I think the error suggests the data passed into the BeautifulSoup method is incorrect: Line 41, in import_temp n_temp = o_temp.text

If anyone is able to tell me the cause of this exception, and how to resolve it, it would be really appreciated.

Thanks

CodePudding user response：

You need to look at the website structure in the browser dev tools. Here's how it looks on this particular site:

<div
  data-testid="FeelsLikeSection"
  
>
  <span
    data-testid="TemperatureValue"
    
    >54°</span
  ><span
    data-testid="FeelsLikeLabel"
    
    >Feels Like</span
  >
</div>

The <span> with real feel temperature has a unique class name, but this class name is auto-generated (probably by something like Styled Components) and will likely change each time the site is rebuilt, so you should not rely on that.

The enclosing <div> on the other hand can be distinguished by the data-testid attribute, which is put there by the site developers for their automated tests. This is unlikely to change, as it will break their tests.

The <span> with the temperature also has the data-testid attribute, but on this site there are many <span>s with the same attribute value, so a query by {'data-testid': 'TemperatureValue'} will not give you a unique element.

So you can do a two-level query like this:

o_temp_feel = soup_in_bowl.find('div', attrs={'data-testid': 'FeelsLikeSection'})
o_temp = o_temp_feel.find('span', attrs={'data-testid': 'TemperatureValue'})
# from the resulting object, "o_temp", get the text parameter and assign it to "n_temp"
n_temp = o_temp.text

Note that the n_temp will be something like "54°", so if you need the numeric value, you need to truncate it before converting to an integer.