Scraping data when there is onclick event button-CodePudding

I am trying to scrape data from "https://lol.inven.co.kr/dataninfo/match/teamList.php" (sorry the website is not in English)

When I don't clikc the orange squre button, the site does not have "detailTable"

When clicking the button, "detailTable" shows up, which I actually want to scrape.

With this dummy webpage

Since it is the dummy webpage, the request URL (POST) does not have any rule.

In this case, is the only way to scrape what appears when clicking onclick button ("detailTable") is Selenium? Is there anyway that I can use BeautifulSoup and Requests combined?

Any ideas are welcome. Thank you in advance!

CodePudding user response：

You can get all the data from that post request but first you need to get the matchcode values.

Then, make a POST request and parse an XML that you get back.

For example:

import re
import time

import requests
from bs4 import BeautifulSoup

headers = {
    "Content-Type": "application/x-www-form-urlencoded;charset=utf-8",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0"
}

with requests.Session() as s:
    soup = (
        BeautifulSoup(
            s.get(
                "https://lol.inven.co.kr/dataninfo/match/teamList.php",
                headers=headers,
            ).text,
            "lxml"
        ).select("#lolDbMatchTeamList .listDetail a")
    )

match_ids = [re.search(r"(\d )", a["onclick"]).group(1) for a in soup]

for match_id in match_ids[:1]:  # just getting one match here
    r = s.post(
        f"https://lol.inven.co.kr/dataninfo/match/match_detail.xml.php?dummy={int(time.time())}",
        data={"matchcode": match_id},
    )
    player_names = (
        "\n".join(
            p.find("playername").getText() for p
            in BeautifulSoup(r.text, features="xml").select("player")
        )
    )
    print(player_names)

This prints player names for the first match.

RNG Ming
RNG Wei
RNG Xiaohu
RNG GALA
RNG Ming
T1 Zeus
T1 Oner
T1 Faker
T1 Gumayusi
T1 Keria

And here's a full XML that you get on each POST request.

<?xml version="1.0" encoding="UTF-8"?>
<resultdata>
    <match>7345</match>
        <player>
            <code>125062</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>900165</playercode><playername>RNG Ming</playername><mname>Ming</mname><champ>155</champ><champicon>Gwen_Square_0</champicon><champname>그웬</champname><spell1icon>lol_spell_ghost</spell1icon><spell1name>유체화</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>3</kill><death>0</death><assist>5</assist><ka>53</ka><kd>3.6</kd><asd>6</asd><kda>9.6</kda>    </player>
        <player>
            <code>125063</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>9002581</playercode><playername>RNG Wei</playername><mname>Wei</mname><champ>34</champ><champicon>LeeSin_Square_0_2</champicon><champname>리 신</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_smite</spell2icon><spell2name>강타</spell2name><kill>2</kill><death>1</death><assist>11</assist><ka>87</ka><kd>2</kd><asd>11</asd><kda>13</kda>    </player>
        <player>
            <code>125064</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>99012</playercode><playername>RNG Xiaohu</playername><mname>Xiaohu</mname><champ>113</champ><champicon>Lissandra_Square_0_1</champicon><champname>리산드라</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>2</kill><death>2</death><assist>10</assist><ka>80</ka><kd>1</kd><asd>5</asd><kda>6</kda>    </player>
        <player>
            <code>125065</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>9002583</playercode><playername>RNG GALA</playername><mname>GALA</mname><champ>64</champ><champicon>Tristana_Square_0_4</champicon><champname>트리스타나</champname><spell1icon>lol_spell_cleanse</spell1icon><spell1name>정화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>5</kill><death>0</death><assist>3</assist><ka>53</ka><kd>6</kd><asd>3.6</asd><kda>9.6</kda>   </player>
        <player>
            <code>125066</code><winloss>1</winloss><side>1</side><teamcode>412</teamcode><teamname>Royal Never Give Up</teamname><playercode>900165</playercode><playername>RNG Ming</playername><mname>Ming</mname><champ>135</champ><champicon>Rakan_Square_0</champicon><champname>라칸</champname><spell1icon>lol_spell_ignite</spell1icon><spell1name>점화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>3</kill><death>0</death><assist>8</assist><ka>73</ka><kd>3.6</kd><asd>9.6</asd><kda>13.2</kda> </player>
        <player>
            <code>125067</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002552</playercode><playername>T1 Zeus</playername><mname>Zeus</mname><champ>100</champ><champicon>Jayce_Square_0_1</champicon><champname>제이스</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_teleport</spell2icon><spell2name>순간이동</spell2name><kill>1</kill><death>4</death><assist>0</assist><ka>33</ka><kd>0.3</kd><asd>0</asd><kda>0.3</kda>  </player>
        <player>
            <code>125068</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002559</playercode><playername>T1 Oner</playername><mname>Oner</mname><champ>80</champ><champicon>MonkeyKing_Square_0_2</champicon><champname>오공</champname><spell1icon>lol_spell_flash</spell1icon><spell1name>점멸</spell1name><spell2icon>lol_spell_smite</spell2icon><spell2name>강타</spell2name><kill>1</kill><death>2</death><assist>2</assist><ka>100</ka><kd>0.5</kd><asd>1</asd><kda>1.5</kda>   </player>
        <player>
            <code>125069</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>135</playercode><playername>T1 Faker</playername><mname>Faker</mname><champ>89</champ><champicon>Ahri_Square_0_2</champicon><champname>아리</champname><spell1icon>lol_spell_cleanse</spell1icon><spell1name>정화</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>0</kill><death>5</death><assist>2</assist><ka>67</ka><kd>0</kd><asd>0.4</asd><kda>0.4</kda>  </player>
        <player>
            <code>125070</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>9002255</playercode><playername>T1 Gumayusi</playername><mname>Gumayusi</mname><champ>129</champ><champicon>Jhin_Square_0</champicon><champname>진</champname><spell1icon>lol_spell_exhaust</spell1icon><spell1name>탈진</spell1name><spell2icon>lol_spell_flash</spell2icon><spell2name>점멸</spell2name><kill>1</kill><death>1</death><assist>1</assist><ka>67</ka><kd>1</kd><asd>1</asd><kda>2</kda>  </player>
        <player>
            <code>125071</code><winloss>2</winloss><side>2</side><teamcode>223</teamcode><teamname>T1</teamname><playercode>900280</playercode><playername>T1 Keria</playername><mname>Keria</mname><champ>144</champ><champicon>yummi_square_0</champicon><champname>유미</champname><spell1icon>lol_spell_exhaust</spell1icon><spell1name>탈진</spell1name><spell2icon>lol_spell_heal</spell2icon><spell2name>회복</spell2name><kill>0</kill><death>3</death><assist>2</assist><ka>67</ka><kd>0</kd><asd>0.7</asd><kda>0.7</kda>    </player>
    </resultdata>