how to fix response.json() no data available error?-CodePudding

Earlier today I was trying to web scrape some climate data from here: https://www.weather.gov/wrh/Climate?wfo=sew. I have since found a better way to get the data I need, but I'm still confused about the code issue.

I'm new to web scraping but have the following code. The session.post() request returns a status code 200, which I assume means the data was found, but the request.json() gives "{'error': 'no data available'}". Does anyone know why this might be?

s = requests.session()
url_search = 'https://data.rcc-acis.org/StnData'

headers = {'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36'}

data = {"elems":[{"name":"maxt","add":"t"},{"name":"mint","add":"t"},{"name":"avgt","add":"t"},{"name":"avgt","normal":"departure91","add":"t"},{"name":"hdd","add":"t"},{"name":"cdd","add":"t"},{"name":"pcpn","add":"t"},{"name":"snow","add":"t"},{"name":"snwd","add":"t"}],"sid":"KRNT 5","sDate":"2022-12-01","eDate":"2022-12-31"}

req = s.post(url_search, headers = headers, json = data)
j = req.json()

CodePudding user response：

Try to pass data in data= parameter, not in json= parameter. Also, try to remove the part from the "sid" parameter:

import pandas as pd
import requests

api_url = "https://data.rcc-acis.org/StnData"

payload = {
    "params": '{"elems":[{"name":"maxt","add":"t"},{"name":"mint","add":"t"},{"name":"avgt","add":"t"},{"name":"avgt","normal":"departure91","add":"t"},{"name":"hdd","add":"t"},{"name":"cdd","add":"t"},{"name":"pcpn","add":"t"},{"name":"snow","add":"t"},{"name":"snwd","add":"t"}],"sid":"OLMthr","sDate":"2022-12-01","eDate":"2022-12-31"}',
    "output": "json",
}

data = requests.post(api_url, data=payload).json()
print(data)

Prints:

{
    "meta": {"state": "WA", "sids": ["OLMthr 9"], "uid": 32815, "name": "Olympia Area"},
    "data": [
        [
            "2022-12-01",
            ["36", 24],
            ["22", 24],
            ["29.0", 24],
            ["-11.2", 24],
            ["36", 24],
            ["0", 24],
            ["0.07", 24],
            ["M", -1],
            ["M", -1],
        ],
        [
            "2022-12-02",
            ["41", 24],
            ["27", 24],
            ["34.0", 24],
            ["-6.0", 24],
            ["31", 24],
            ["0", 24],
            ["0.43", 24],
            ["M", -1],
            ["M", -1],
        ],

...and so on.