I have a problem clicking every button on the LinkedIn page. In some profiles which contain a lot of information about job experience, schools, license we have to expand this information by click on 'Show more button".

Sample profile 1

Sample profile 2

I try many things like searching for elements by Xpath and then looping them to click every button on the page but it didn't work - because every button class is the same as other elements that we can find using selenium. I figure it that first "show more" button is always for experiane section and that code make job to click it:

self.driver.execute_script("arguments[0].click();", WebDriverWait(
            self.driver, 3).until(EC.element_to_be_clickable((By.XPATH, "//li-icon[@class='pv-profile"
                                                                        "-section__toggle-detail-icon']"))))

Then we have the education section, license, and certification section - this makes me trouble. Temporary solution is to click at element that contain string:

self.driver.find_element_by_xpath("//*[contains(text(), 'Pokaż więcej')]").click()

OR

self.driver.find_element_by_xpath("//*[contains(text(), 'Pokaż 1 uczelnię więcej')]").click()

Sooner than later I know that code has a lot of limitations. Does anyone have a better idea of how to solve this problem?

Solution

containers = self.driver.find_elements_by_xpath("//li-icon[@class='pv-profile-section__toggle-detail-icon']")
    for button in containers:
        self.driver.execute_script('arguments[0].click()', button)

CodePudding user response：

I tested page with own code and it seems you can get all buttons with

items = driver.find_elements(By.XPATH, '//div[@]//button')

for item in items:
    driver.execute_script("arguments[0].click();", item)

But there can be other problem. Page uses "lazy loading" and it may need to use JavaScript code which scrolls down to load all component.

Here is my full code with some ideas in comments.

I tried also select buttons in sections but not all methods work.
But maybe it will be useful for other ideas.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException, TimeoutException
import time

USERNAME = 'XXXXX'
PASSWORD = 'YYYYY'

url = 'https://www.linkedin.com/in/jakub-bialoskorski/?miniProfileUrn=urn:li:fs_miniProfile:ACoAABp5UJ8BDpi5ZwNGebljqDlYx7OXIKgxH80'

driver = webdriver.Firefox()

# -------------------------------------

driver.get(url)
time.sleep(5)

#wait = WebDriverWait(driver, 10)

cookies = driver.find_element(By.XPATH, '//button[@action-type="ACCEPT"]')
cookies.click()
time.sleep(1)

link = driver.find_element(By.XPATH, '//p[@]/button')
link.click()
time.sleep(1)

login_form = driver.find_element(By.XPATH, '//div[@]')
time.sleep(1)

username = login_form.find_element(By.XPATH, '//input[@id="session_key"]')
username.send_keys(USERNAME)
password = login_form.find_element(By.XPATH, '//input[@id="session_password"]')
password.send_keys(PASSWORD)
time.sleep(1)

#button = login_form.find_element(By.XPATH, '//button[@type="submit"]')
button = login_form.find_element(By.XPATH, '//button[contains(text(), "Zaloguj się")]')
button.click()
time.sleep(5)

# -------------------------------------

url = 'https://www.linkedin.com/in/jakub-bialoskorski/?miniProfileUrn=urn:li:fs_miniProfile:ACoAABp5UJ8BDpi5ZwNGebljqDlYx7OXIKgxH80'

#from selenium.webdriver.common.action_chains import ActionChains

driver.get(url)
time.sleep(5)

# -----------

print('... scrolling for lazy loading ...')
     
last_height = 0
while True:
    
    driver.execute_script("window.scrollTo(0, window.scrollY   window.innerHeight);")
    time.sleep(2)    

    new_height = driver.execute_script("return window.scrollY")
    if new_height == last_height:
        break
    last_height = new_height      

# -----------

def click_items(items):
    for item in items:
        print('text:', item.text)

        #print(item.get_attribute('innerHTML'))

        #print('... scrolling ...')
        #ActionChains(driver).move_to_element(item).perform()
       
        print('... scrolling ...')
        driver.execute_script("arguments[0].scrollIntoView(true);", item)
        
        #print('... clicking ...')
        #item.click()
        #time.sleep(1)

        print('... clicking ...')
        driver.execute_script("arguments[0].click();", item)
        time.sleep(1)
        
        print('----')
   
print('\n>>> Pokaż <<<\n')

#items = driver.find_elements(By.XPATH, '//button[contains(text(), "Pokaż")]')
#click_items(items)

print('\n>>> Doświadczenie - Pokaż więcej <<<\n')

#section = driver.find_elements(By.XPATH, '//section[@id="experience-section"]')
#items = driver.find_elements(By.XPATH, '//button[contains(text(), "zobacz wi")]')
items = driver.find_elements(By.XPATH, '//button[contains(@class, "inline-show-more-text__button")]')
click_items(items)

print('\n>>> Umiejętności i potwierdzenia - Pokaż więcej  <<<\n')

#section = driver.find_elements(By.XPATH, '//section[@id="experience-section"]')
items = driver.find_elements(By.XPATH, '//button[@data-control-name="skill_details"]')
click_items(items)

print('\n>>> Wyświetl <<<\n')

items = driver.find_elements(By.XPATH, '//button[contains(text(), "Wyświetl")]')
click_items(items)

print('\n>>> Rekomendacje <<<\n')

items = driver.find_elements(By.XPATH, '//button[@aria-controls="recommendation-list"]')
click_items(items)

print('\n>>> Osiągnięcia <<<\n')

print('--- projects ---')
items = driver.find_elements(By.XPATH, '//button[@aria-controls="projects-expandable-content"]')
click_items(items)

print('--- languages ---')
items = driver.find_elements(By.XPATH, '//button[@aria-controls="languages-expandable-content"]')
click_items(items)

# --- all buttons ---
#items = driver.find_elements(By.XPATH, '//div[@]//button')
#click_items(items)