Gitlab CI & Django: How to install custom package with pip-CodePudding

I have a Django project that have many dependencies and among those are several custom private Django package listed in our requirements.txt file at the project root.

I want to setup simple CI that triggers our tests each time a commit is made.
To do so I have written a simple .gitlab-ci.yaml file that tries to run those tests but I am having trouble installing our custom dependencies.
They are listed in our requirements like follow:

...
Django==3.2.12
...
-e git ssh://[email protected]/{organization}/{project}.git@{commit-sha}#egg={project}
-e git ssh://[email protected]/{organization}/{project}.git@{{commit-sha}#egg={project}
...

Note: All the mentionned projects lies under the same Gitlab organization

Here is what my .gitlab-ci.yaml file looks like:

stages:
  - test

run-test: 
  image: ubuntu:18.04
  stage: test
  before_script: # installing python, pip & installing requirements
    - apt -y update

    - apt -y install apt-utils git net-tools
    - apt -y install python3.8 python3-pip
    - apt -y upgrade
    
    - python3 -m pip install --upgrade pip
    - cd servers/api
    - pip3 install -r ../requirements.txt
  script:
    - python3 manage.py test

This obviously fails giving the following error:

Obtaining {project} from git ssh://****@gitlab.com/{organization}/{project}.git@{commit-sha}#egg={project} (from -r ../requirements.txt (line 32))
Cloning ssh://****@gitlab.com/{organization}/{project}.git (to revision {commit-sha}) to ./src/{project}
Running command git clone --filter=blob:none -q 'ssh://****@gitlab.com/{organization}/{project}.git' /builds/{organization}/platform/servers/api/src/{project}
  Host key verification failed.
  fatal: Could not read from remote repository.
  Please make sure you have the correct access rights
  and the repository exists.

Reading this topic from the Gitlab doc I have tried adding SSH key in the mix but it did not work either.

I have also found this Gitlab issue that seems to talk about the same topic but it requires to create PyPi private package and I am not quite sure how to do it neither if I should

Any help is appreciated

CodePudding user response：

Fixing pip install over ssh

If you want to continue using ssh to install with pip, you'll need to fix the ssh host key verification issue.

Host key verification failed.

You can fix this issue by setting GIT_SSH_OPTIONS to ignore host key verification.

before_script:
  - export GIT_SSH_COMMAND="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

Of course, this is not ideal since you're no longer verifying the identity of the git server.

Alternatively if you don't want to skip host key verification, you can verify the host keys, as described here and add the host key for your server to the known_hosts file.

You might also avoid host key issues altogether by using HTTPS instead of ssh, using HTTP basic auth with pip. That is to say use git https instead of git ssh.

Use the package registry (recommended!)

As mentioned in the post you found in your question, GitLab has a PyPI package registry that allows you to publish Python packages as well as use them with pip. This will require you to publish the packages and setup the (additional) index url(s) in your pip configuration. The documentation covers the setup and usage.