Fetch dynamically generated website content and write as text file?-CodePudding

I want to store this site as text file:

https://www.tradingview.com/symbols/BTCUSD/technicals/

But I have a hard time since it's content is dependent on "buttons" that generate the content dynamically.

I wrote this code:

import requests
from bs4 import BeautifulSoup
results = requests.get("https://www.tradingview.com/symbols/BTCUSD/technicals/")
src = results.content
soup = BeautifulSoup(src)

var = soup.find_all()

with open('readme.txt', 'w') as f:
    f.write(str(var))

But it appears it only fetches the source code and not the dynamically generated content. E.g. when clicking the button "1 minute" the content changes. I want some sort of "snapshot" of what "1 minute" generates as content and then sorta look for keywords "buy" or "sell" somehow later.

Really stuck with the first step of fetching the website's dynamic content... Can anyone help?

CodePudding user response：

If you want to be able to get different content based on which buttons you click, you'll want to use Selenium instead of BeautifulSoup.

Here's a rough example which uses the page you provided:

from selenium import webdriver
from selenium.webdriver.common.by import By
import os

driver_path = os.path.abspath('chromedriver')
driver = webdriver.Chrome(driver_path)
driver.get('https://www.tradingview.com/symbols/BTCUSD/technicals/')

driver.implicitly_wait(10)

# Get the '1 minute' button element
btn_1min = driver.find_element(By.XPATH, '//*[@id="1m"]')
# Get the '15 minutes' button element
btn_15min = driver.find_element(By.XPATH, '//*[@id="15m"]')

# Decide which button to click on based on user input
choice = ''

while True:
    time_input = input('Choose which button to click:\n> ')

    if time_input == '1':
        btn_1min.click() # Click the '1 minute' button
        choice = '1 min'
        break
    elif time_input == '15':
        btn_15min.click() # Click the '15 minute' button 
        choice = '15 min'
        break
    else:
        print("Invalid input - try again")

# Get the 'Summary' section of the selected page
summary = driver.find_element(By.XPATH, '//*[@id="technicals-root"]/div/div/div[2]/div[2]')

# Get the element containing the signal ("Buy", "Sell", etc.)
signal = summary.find_element(By.XPATH, '//*[@id="technicals-root"]/div/div/div[2]/div[2]/span[2]')

# Get the current text of the signal element
signal_txt = signal.text

print(f'{choice} Summary: {signal_txt}')

driver.quit()

This script gives us the option to choose which button to click via user input, and prints different content based on which button was chosen.

For example: If we input '1' when prompted, thereby telling the script to click the "1 minute" button, we get the signal currently displayed in the Summary section of this button's page, which in this case happens to be "BUY":

Choose which button to click:
> 1
1 min Summary: BUY

If you want to read up on how to use Selenium with Python, here's the documentation: https://selenium-python.readthedocs.io