I have troubles understanding Pythons package management and virtual environments. So far I managed to create virtual environments on cmd line and using PyCharm.

But i am still confused when actually working with them:

When i use pip search yaml in the project root folder after activating the (venv) to lookup packages i get the following message:

ERROR: XMLRPC request failed [code: -32500]
RuntimeError: PyPI's XMLRPC API is currently disabled due to unmanageable load and will be 
deprecated in the near future. See https://status.python.org/ for more information.

And i can not interpret the information i get on the mentioned website. The question is now how can i search for Python packages with pip? And if pip does not work, which tool is to be used instead? I would expect such information rather than the information given on the ERROR or on the mentioned website. pip itself does not mention some other API in its documentation.

A app i use imports a package named "yaml". When i am trying to search for such package (using PyCharm), i get a message like:

ERROR: Could not find a version that satisfies the requirement yaml (from versions: none)
ERROR: No matching distribution found for yaml

But why can i then lookup such package in PyCharm (Settings/Project/Python Interpreter)?

Another Python library the app uses is PIL. Same issue as with yaml. I can lookup the package using PyCharm, it shows a package information banner talking about Python Image Library, version 1.1.6, but when i want to install it PyCharm tells that there is none such library:

ERROR: Could not find a version that satisfies the requirement PIL (from versions: none)
ERROR: No matching distribution found for PIL

I looked at PyPi.org, and a search for PIL shows 1.029 packages. Which one should i take? There are tons of packages in the result which do not even mention PIL in their names or descriptions. Is this useful?

I am using Python 3.8.10 now as before i installed 3.10 and 3.9 and i thought maybe the packages are not yet ported to the newer python releases. But even on 3.8 they seem not to be available.

Which brings me to another question:

When i installed a Python release and created a virtual environment from that Python release, is pip or whatever tool i use to install packages for that venv aware for which Python release the package should be installed? So that the package can be used with the Python release and is compatible with it? Or do i need to take care about this by myself and if so, how can i ensure this? As far as i can see in the packages detail info on PyCharm there is no information if it is compatible with the Python release used in the venv.

And finally, how can i use virtual environments when thinking about deploying a Python app.

During development e.g. using PyCharm i commit the stuff i wrote to git. Should i commit the venv to git as well? And how do i roll out the Python app? Just copy over the complete project folder to the target system, or check it out using git? Including the venv folder? Should i install the exact same version of the Python interpreter first on the target system? How can i then run the app to ensure that all the needed packages which are installed in venv are used?

CodePudding user response：

Since you mention you use PyCharm: in Pycharm you can go to Settings -> Project -> Interpreter to add and remove packages. In the window to add packages you can filter by name and pick specific versions.

As mentioned in the comments, you can also search for packages on pypi.org.

Another way to add packages, which is arguably recommended, is to create a requirements.txt file (the name is conventional but can be changed), populating it with one package per line, for which you can specify specific version (ranges) to install. Once you have a requirements.txt file, with your virtual environment activated you can run pip install -r requirements.txt to install all packages from it.

You can create a requirements.txt from your currently installed packages by running pip freeze > requirements.txt.

CodePudding user response：

first of all, I'm honestly not familiar with pip so I can't help you with that one much. However, when it comes to the other part of your question about Python virtual environment and packages, I can definitely help you with that as they're something I have to deal with at my workplace.

I'm going to break down this writing into several sections under different headings so here it goes :)

Python Virtual Environment with `Mamba`/`Conda`

Essentially, a python virtual environment is a directory containing a specific python version and all packages you want to use for a specific Python project. At my workplace, I'm using mamba and conda package manager instead of pip.

`base` Virtual Environment

When you install mamba or conda, it's going to create a base virtual environment that contains only Python and all of its basic packages.

On Windows, if you activate this base environment and you type python, it will execute the Python executable under

C:\Users\<user_name>\mambaforge\python.exe if you're using mamba package manager
C:\Users\<user_name>\Anaconda3\python.exe if you're using conda package manager

From my experience, I highly recommend mamba package manager instead of conda because is much faster when looking for packages and resolving dependencies for a package we want to install. mamba is essentially an improved version of conda package manager and it's much faster because it's written with C . Please look at this github page for more information on mamba

Adding New Virtual Environment

Now, suppose that you create a new virtual environment called new_environment by running conda create --name environment.

This will create a new directory where the Python executable and all of its packages are going to be installed to. On Windows, this directory is going to be created at this path:

C:\Users\<user_name>\mambaforge\envs\new_environment\ if you're using mamba package manager
C:\Users\<user_name>\Anaconda3\envs\new_environment\ if you're using conda package manager

Searching for Packages

As @navneethc said, pip has unfortunately disabled their search command. From what I understand, looks like it's too costly for them to enable this feature and process all the requests on their server.

However, good news, with mamba/conda, you can still search for packages by typing this on your command line/terminal:

conda search <package_name> --channel <channel_name> if you're using conda
mamba repoquery search <package_name> --channel <channel_name> if you're using mamba

The command will then output a list of all versions of the package to your command line/terminal.

Installing Packages

Now, from the command line, in order to install packages to this new_environment, we just need to run:

conda activate new_environment
conda install <package_name> --channel <channel_name>

if you're using mamba, replace conda with mamba above

Now, when you install Python to this new_environment, the python executable is going to be located at

C:\Users\<user_name>\mambaforge\envs\new_environment\python.exe if you're using mamba to install Python
C:\Users\<user_name>\Anaconda3\envs\new_environment\python.exe if you're using conda to install Python

Again, when you activate new_environment and run python, it's going to use the python executable located at this location.

All other packages (except for Python) are going to be installed under the site-packages directory:

C:\Users\<user_name>\Anaconda3\envs\new_environment\site-packages\

--NOTE--

You asked if pip will automatically install the proper package version and build for the specific python version installed in the activated virtual environment. I'm not too sure about pip but for conda/mamba, they will pin the python version and conda/mamba will install the appropriate package version/build which corresponds to your python version automatically

Export Environment to a `yaml` File

In order for you to save your virtual environment to a Git repository hosting service like GitHub, you need to export your virtual environment to a yaml file by running this command:

conda env export -f <name_of_yaml_file>.yaml

You must then add this yaml file and then commit it to your Git repository, then push the commit to GitHub.

Deploying Python App

I don't know what kind of Python app you're developing. For my scripts, I simply just cloned the git repository from BitBucket (my workplace uses BitBucket) on the target machine. Whenever modification is made to my python script on my local machine, I commit and push that change and then do git pull on the target machine.

Re-create Virtual Environment on Target Machine

First, you want to install conda/mamba (again, mamba is highly recommended) on your target machine.

Then, re-create the virtual environment on the target machine with the help of the yaml file that you created. You can easily do this by going to your project directory, and then run conda env create -f <name_of_yaml_file>.yaml

You can then activate this environment and run your Python app inside this environment

Deploy Key/Access Key

I was told that the proper way of this is to actually automate the process of git pull using something called deploy key on GitHub or access key on BitBucket.

With This special ssh-key, apparently we can define a git workflow using some kind of script which will do the git pull on the target machine whenever something happens in our remote repository. For instance, we can define the script to be executed whenever a feature branch is merged to our main branch

My knowledge in this area is definitely limited. If someone is more familiar with this, please go ahead and correct me.

Package Python App and Host It on Custom Channel

I'm still learning about this myself at my workplace but it looks like we can build our Python app into a package and then host this package in our own custom channel. Then, you can simply just do

conda install <our_app_package_name> --channel <our_custom_channel>

on the target machine (after activating our virtual environment) to install our package to the target machine.

There's this guide written by python.org on how to package our Python projects and I'm still reading this one myself

Again, feel free to correct what I said if it's somehow incorrect but just sharing my experience with Python package management with you @Joysn!

Python Virtual Environment with Mamba/Conda

base Virtual Environment