Selenium (Python) (Firefox) is unable to write Firefox profile. Permission denied (os error 13)-CodePudding

When I run my Python code using Selenium to open (and scrape) a website, using a profile parameter, I get the following error message:

    selenium.common.exceptions.SessionNotCreatedException: Message: Failed to set preferences: Unable to write Firefox profile: Permission denied (os error 13)

I use the following code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from selenium.webdriver.firefox.options import Options
from datetime import datetime

f = open("/home/username/Documents/AH/log2.txt", "a")
f.write('gestart')
f.close()

# I use the below code when I use crontab, read below..
#from pyvirtualdisplay import Display
#display = Display(visible=0, size=(2880, 1800)).start()

website = "https://www.ah.nl/"
options=Options()
options.add_argument('-profile')
options.add_argument(r'/home/username/.mozilla/firefox/n2n9yy4r.default-release') 

driver = webdriver.Firefox(options=options, service_log_path="/home/username/geckodriver.log")

driver.implicitly_wait(20)
driver.get(website)

tijd = WebDriverWait(driver,40).until(EC.presence_of_element_located((By.XPATH,"//*[@id='navigation-header']/div[1]/div[1]/div/div/p/a")))
tijdbezorging = tijd.text

driver.close()
driver.quit()

f = open("/home/username/Documents/AH/testlog.txt", "a")
f.write(tijdbezorging)
f.close()

The full error message when I run this code using python3 filenamecode.py:

SendAHStatus.py:46: DeprecationWarning: service_log_path has been deprecated, please pass in a Service object
  driver = webdriver.Firefox(options=options, service_log_path="/home/username/geckodriver.log")
/usr/lib/python3/dist-packages/requests/__init__.py:89: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!
  warnings.warn("urllib3 ({}) or chardet ({}) doesn't match a supported "
Traceback (most recent call last):
  File "SendAHStatus.py", line 46, in <module>
    driver = webdriver.Firefox(options=options, service_log_path="/home/username/geckodriver.log")
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/firefox/webdriver.py", line 197, in __init__
    super().__init__(command_executor=executor, options=options, keep_alive=True)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 288, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 381, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/webdriver.py", line 444, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.8/dist-packages/selenium/webdriver/remote/errorhandler.py", line 249, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: Failed to set preferences: Unable to write Firefox profile: Permission denied (os error 13)

The code DOES work when I schedule it using sudo crontab -e. Yes, using sudo. See code below:

SHELL=/bin/bash
PATH=/usr/local/bin/:usr/bin: and some paths
01 01 * * * python3 myscriptname.py

And, the code also works (via my command line) without the profile parameter. I also tried using a different profile path like, /home/username/profiledir. This also works via sudo crontab, but not via my regular command line.

--ADDED---

I created the profile in FireFox using about:profiles -> create new profile. From there I gave all users, incl. the user that I am using 'odroid', all permissions to this folder, pls see below:

---//ADDED------

Btw, I uncomment the virtual display when using cron.

I could run my code using sudo python3 myscript.py. But then I get this message:

Running Firefox as root in a regular user's session is not supported.  ($XAUTHORITY is /home/username/.Xauthority which is owned by username.)

When changing ownership of XAUTHORITY to root, I can run my script, but I don't want to go down that path.

What I can derive from all of this:

The webdriver is working,
Its a permission issue since the code runs using elevated permission, using cron.

Please help me with this, its getting frustrating.

My environment:

Selenium 4.6.0
Python 3.8
Firefox 107.0
Ubuntu 20.04.5 LTS

Your help is much appreciated!

CodePudding user response：

The problem is that the FirefoxProfile class was not properly updated from the switch to geckodriver. Back when we used to have the Firefox driver and it used the extension, it was managed by FirefoxProfile class, so it needed to set default values from a manifest. This is no longer needed.

Instead of just zipping the provided directory, the existing code wants to update things, and whatever it is doing to update them is breaking it. If this line gets commented out - check this

Working code:

options = webdriver.FirefoxOptions()
options.profile = "/Users/titusfortner/Library/ApplicationSupport/Firefox/Profiles/1amwoj91.testing"
driver = webdriver.Firefox(options=options)