How to run shell-exectued python script in a virtual environment-CodePudding

I have a shell script that executes some python code, and installs the necessary dependencies to do so beforehand:

sudo -E PATH="$PATH" python3 -m pip install -r requirements.txt

The script runs successfully when executed like so:

./script.sh

but I get an import error from Python when I run the script with sudo permission:

sudo ./script.sh

Traceback (most recent call last):
  File "dsc.py", line 2, in <module>
    from imblearn.over_sampling import ADASYN
  File "/usr/local/lib/python3.8/dist-packages/imblearn/__init__.py", line 53, in <module>
    from . import ensemble
  File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/__init__.py", line 8, in <module>
    from ._forest import BalancedRandomForestClassifier
  File "/usr/local/lib/python3.8/dist-packages/imblearn/ensemble/_forest.py", line 28, in <module>
    from sklearn.utils.fixes import _joblib_parallel_args
ImportError: cannot import name '_joblib_parallel_args' from 'sklearn.utils.fixes' (/usr/local/lib/python3.8/dist-packages/sklearn/utils/fixes.py)

I don't understand why Python can no longer import ADASYN when the script is given sudo permission.

I am using virtualenv for version control. When I check where the imblearn package is located when in the same virtual environment from where the shell script was exectued, I get:

❯ python3 -m pip show imblearn
WARNING: Package(s) not found: imblearn

But I can find it when not in a virtual environment:

❯ deactivate
❯ python3 -m pip show imblearn
Name: imblearn
Version: 0.0
Summary: Toolbox for imbalanced dataset in machine learning.
Home-page: https://pypi.python.org/pypi/imbalanced-learn/
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: /usr/local/lib/python3.8/dist-packages
Requires: imbalanced-learn
Required-by:

Which means that my shell script does install the dependencies, but not within the virtual environment from which it is executed. Removing the imblearn package and running the shell script installs it again, so I am certain that it is this shell script that installs the package to the default location. I thought that by running Python code with the sudo -E PATH="$PATH" option makes sure that the python command has the same path as the virtual environment from which it is run, and thus the correct dependencies. But given that the dependencies get installed in my default Python package directory /usr/local/lib/python3.8/dist-packages, this seems not to be the case.

So how can I run python code in a shell script within the same virtual environment from which the shell script was executed? Is this possible at all? I am familiar with Python, but very new to shell.

I'm using Python 3.8.10 (both as default and in the virtualenv), Ubuntu 20.04 and zsh.

CodePudding user response：

If you run sudo ./script.sh, the actually executor of the script is the superuser, which means none of the env-related stuff in your virtual env is synced over.

You may need to specify the python virtual env inside your ./script.sh, i.e., adding one line on the top of the ./script.sh: #!path/to/your/virtualenvs/python

CodePudding user response：

The problem can be mitigated by running the bash script like so:

sudo -E PATH=$PATH ./script.sh

instead of just sudo ./script.sh.

This way, sudo exectues the shell script, and the python path of your current virtualenv is passed along.

Inside the shell script, you now do not need to execute all python scripts with sudo -E PATH=$PATH script.py, but simply using python script.py suffices, since in this case python refers to the virtualenv python version.