Error trying to save scraped web data to a text file-CodePudding

I am new, and recently started using Python. I am trying to save the retrieved Twitter followers from web to a text file but it doesn't work.

Here's my code:

for twusernames in driver.find_elements_by_xpath('//div[@aria-label="Timeline: Followers"]//a[@role="link"]'):
    print(twusernames.get_property('href'))
    file = open('links.txt', 'w')
    file.write(twusernames.get_property('href'))
    file.close()

What am I doing wrong? :( Thanks for help.

CodePudding user response：

This should work:

with open('links.txt', 'w') as file
for twusernames in driver.find_elements_by_xpath('//div[@aria-label="Timeline: Followers"]//a[@role="link"]'):
    content = twusernames.get_property('href')
    print(content)
    file.write(content)
file.close()

CodePudding user response：

Try this.

with open('links.txt', 'w') as file
    file.write(twusernames.get_property('href'))
    file.close()

CodePudding user response：

get_attribute()

get_attribute() gets the given attribute or property of the element. This method first tries to return the value of a property with the given name. If a property with that name doesn’t exist, it returns the value of the attribute with the same name. If there is no attribute with that name, None is returned. Values which are considered truthy, that is equals "true" or "false", are returned as booleans. All other non-None values are returned as strings. For attributes or properties which do not exist, None is returned. To obtain the exact value of the attribute or property you can also use get_dom_attribute() or get_property() methods.

Additionally, once you open a file_handle you need to close it. However, file operations being I/O operations must be as less as possible. So effectively your optimized code block will be:

Using get_attribute():

file = open("links.txt", "w")
for twusernames in driver.find_elements_by_xpath('//div[@aria-label="Timeline: Followers"]//a[@role="link"]'):
    file.write(twusernames.get_attribute("href")   "\n")
file.close()

Using get_property():

file = open("links.txt", "w")
for twusernames in driver.find_elements_by_xpath('//div[@aria-label="Timeline: Followers"]//a[@role="link"]'):
    file.write(twusernames.get_property("href")   "\n")
file.close()

Using get_dom_attribute():

file = open("links.txt", "w")
for twusernames in driver.find_elements_by_xpath('//div[@aria-label="Timeline: Followers"]//a[@role="link"]'):
    file.write(twusernames.get_dom_attribute("href")   "\n")
file.close()