pip install custom package from BitBucket with SSH without entering SSH password during Docker build-CodePudding

I'm trying to pip install (via a conda env create command) a custom python package from BitBucket during a docker build command without entering the SSH password/passphrase. This question is similar to this other question, but different because the error is occurring during the docker build command.

The environment.yml file for the conda env create command (during the docker build) looks something like this:

name: my_app
channels:
  - defaults
dependencies:
  - pip=21.2.2
  - python=3.8.11
  - pip:
    - git ssh://[email protected]/my_org/my_package_repo.git
    - pandas==1.2.5
    - python-dotenv==0.19.0
    - xlrd==2.0.1
prefix: /usr/local/anaconda3/envs/my_app

When the docker build is trying to build the conda environment inside the Docker image, I get this error:

Installing pip dependencies: ...working... Pip subprocess error:
  Running command git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw
  Host key verification failed.
  fatal: Could not read from remote repository.

  Please make sure you have the correct access rights
  and the repository exists.
WARNING: Discarding git ssh://****@bitbucket.org/my_org/my_package_repo.git. Command errored out with exit status 128: git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw Check the logs for full command output.
ERROR: Command errored out with exit status 128: git clone -q 'ssh://****@bitbucket.org/my_org/my_package_repo.git' /tmp/pip-req-build-3t5mkmnw Check the logs for full command output.

Ran pip subprocess with arguments:
['/opt/conda/envs/work_content/bin/python', '-m', 'pip', 'install', '-U', '-r', '/tmp/condaenv.9p_rq9_h.requirements.txt']
Pip subprocess output:
Collecting git ssh://****@bitbucket.org/my_org/my_package_repo.git (from -r /tmp/condaenv.9p_rq9_h.requirements.txt (line 3))
  Cloning ssh://****@bitbucket.org/my_org/my_package_repo.git to ./pip-req-build-3t5mkmnw

failed

CondaEnvException: Pip failed

This remote/custom package installs successfully when I'm in my terminal and follow the answer in the referenced StackOverflow question. However, when I try to same thing before running the docker build command, I get the error above. I'm assuming that's because Docker is building a whole new OS image, and it no longer has the SSH RSA passphrase I provided in the terminal. How can I get past this error without providing the passphrase during the build?

Update

Based on suggestions in one of the current answers, I've modified my Dockerfile to look something like this:

# syntax=docker/dockerfile:experimental
FROM continuumio/miniconda3:4.10.3
...
RUN apt-get install -y openssh-client git
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan bitbucket.org >> ~/.ssh/known_hosts
RUN --mount=type=ssh git clone [email protected]:my_org/my_package_repo.git /tmp/my_package_repo

The --mount command creates a local copy of the repository in the /tmp directory. So now the conda environment.yml file needs to be changed to pip install from that local directory like this:

name: my_app
channels:
  - defaults
dependencies:
  - pip=21.2.2
  - python=3.8.11
  - pip:
    - /tmp/my_package_repo
    - pandas==1.2.5
    - python-dotenv==0.19.0
    - xlrd==2.0.1
prefix: /usr/local/anaconda3/envs/my_app

... and I'm running the docker build process with commands like these:

eval $(ssh-agent); ssh-add ~/.ssh/id_rsa
DOCKER_BUILDKIT=1 docker build --ssh default .

The eval command above is to manually input my SSH passphrase before the Docker build begins. The DOCKER_BUILDKIT=1 in front of the docker build command forces Docker to build with Docker Buildkit which is required for the RUN --mount=type=ssh git clone command in the Dockerfile. This solution works for me now. It's not exactly what's in the answer below so I thought I'd share this with the community. I'll be marking the answer that pointed me in this direction as the correct answer.

CodePudding user response：

SSH needs access to your private RSA key for the authentication. Your private key is not shared with any command which runs inside a Docker image. Docker has a builtin feature to solve this issue, which is documented here.

Essentially what you want to do is to download your PUBLIC key to the Docker image:

# Download public key for github.com
RUN mkdir -p -m 0600 ~/.ssh && ssh-keyscan github.com >> ~/.ssh/known_hosts

Define a mount with type ssh in order for Docker Engine to forward SSH agent connections.

RUN --mount=type=ssh git clone [email protected]:myorg/myproject.git myproject

This run command can be something else, for example something which installs your Pythin packages.

You build your docker image with -ssh option:

docker build --ssh default .