Apx_roots tags are empy. How to fix this?-CodePudding

I'm scraping this website - https://registry.verra.org/app/projectDetail/VCS/3651 But unfortunately apx_root tags are empty. How can I solve this problem? Here's my Code:

import requests
from bs4 import BeautifulSoup
import time

source = requests.get('https://registry.verra.org/app/projectDetail/VCS/3651').text
time.sleep(10)
soup = BeautifulSoup(source, 'lxml')
print(soup)
info_gathered = {'Heading': [], 'ID': [], 'Category': [], 'Proponent': [],
                 'Project Status': [], 'EAER': [], 'Project Type': [],
                 'Methodology': [], 'Project Validator': [], 'CPT': [],
                 'File_link': []}
headline = ''
for div in soup.find_all('div'):
    print(div)
    headline = div['card-header bg-primary']
    print(headline)

And in the result I get empty apx-root tags.

CodePudding user response：

The data you see is loaded via JavaScript from external URL. To download the data you need only know the the ID (3651 in this case):

import json
import requests

api_url = "https://registry.verra.org/uiapi/resource/resourceSummary/3651"
data = requests.get(api_url).json()

# pretty print the data:
print(json.dumps(data, indent=4))

Prints:

{
    "resourceIdentifier": "3651",
    "resourceName": "Ege Biogas Project",
    "description": "Ege Biogas project (hereafter referred to as the project) is a biogas to-energy project that will generate renewable energy by capturing biogas from animal manure -via anaerobic digestion- and utilizing it to produce thermal and electric energy through biogas systems. The project is being implemented by EGE B\u0130YOGAZ ELEKTR\u0130K \u00dcRET\u0130M ANON\u0130M \u015e\u0130RKET\u0130, as the Project Owner, in \u00c7apakl\u0131 quarter, Salihli district, Manisa province, in Turkey. In the project's final environmental impact assessment report, it is estimated that approximately 400 tons of cattle manure and chicken hens generated at animal farms will be collected daily or every two days through special sewage trucks equipped with close-tanks to prevent any odor and/or manure leakages and to be fed into the anaerobic digesters at the proposed biogas power plant. The proposed project activity has two biogas engines installed at the biogas power plant, with 3.120 MWe. The manure produced by 40 neighboring farms is collected and fed into the anaerobic digesters at the project site. The project began to generate electricity on April 23rd, 2021, which is the date of gas engines had implemented. According to Electricity Generation License, the biogas generator system has the maximum operational capacity 3.120 MWe. The project has obtained the approval of Environmental Impact Assessment (EIA) issued by the Ministry of Environment and Urbanization in Turkey on January 9th, 2020, and the Electricity Generation License (EGL) issued by the Energy Market Regulatory Authority (EMRA) in Turkey on December 10th, 2020. This project adopts the renewable crediting period of 7 years, twice renewable for 21 years. The expected average annual emission reductions are 322,680 tCO2eq/y. Accordingly, the project is expected to generate 2,258,762 tCO2eq emissions reduction during the first crediting period and 6,776,286 tCO2eq emissions reduction throughout its three crediting periods for a total of 21 years.",
    "location": {
        "latitude": 38.56868,
        "longitude": 28.198723
    },
    "inPublicCommentPeriod": true,
    "lastPublicCommentPeriod": null,
    "attributes": [
        {
            "code": "PROJECT_ID",
            "values": [
                {
                    "type": "string",
                    "value": "3651"
                }
            ]
        }
    ],
   

...

EDIT: The summary info is stored under key participationSummaries:

print(data["participationSummaries"])

Prints:

[
    {
        "programCode": "VCS",
        "attributes": [
            {
                "code": "PROPONENT_NAME",
                "values": [
                    {
                        "type": "string",
                        "value": "Ege Biyogaz Elektrik Üretim Anonim Şirketi",
                    },
                    {"type": "string", "value": "Ankara, Turkey"},
                ],
            },
            {
                "code": "PROJECT_STATUS",
                "values": [{"type": "string", "value": "Under validation"}],
            },
            {
                "code": "EST_ANNUAL_EMISSION_REDCT",
                "values": [{"type": "string", "value": "322680"}],
            },
            {
                "code": "PRIMARY_PROJECT_CATEGORY_NAME",
                "values": [
                    {"type": "string", "value": "Waste handling and disposal"}
                ],
            },
            {
                "code": "PROTOCOL_NAME",
                "values": [{"type": "string", "value": "AM0073"}],
            },
            {
                "code": "VALIDATOR_NAME",
                "values": [
                    {
                        "type": "string",
                        "value": "Earthood Services Private Limited",
                    }
                ],
            },
            {
                "code": "CREDIT_PERIOD_INFO",
                "values": [
                    {"type": "string", "value": "1st, 23/04/2021 - 22/04/2028"}
                ],
            },
        ],
    }
]