I'm trying to make a https request from within a docker container. Here's python code that runs fine on my Windows 10 host:
import certifi
import ssl
import urllib.request
tmp_filename = "penguin.jpg"
pingu_link = "https://i.pinimg.com/originals/cc/3a/1a/cc3a1ae4beafdd5ac2293824f1fb0437.jpg"
print(certifi.where())
default = ssl.create_default_context()
https_handler = urllib.request.HTTPSHandler(context=ssl.create_default_context())
opener = urllib.request.build_opener(https_handler)
# add user agent headers to avoid 403 response
opener.addheaders = [
(
"User-agent",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0",
)]
urllib.request.install_opener(opener)
r = urllib.request.urlretrieve(pingu_link, tmp_filename)
If I understand correctly, certifi comes with its own set of CA certificates, which are contained in the .pem file you can find by calling certifi.where(). However, if I convert this file to .crt and tell request to use it by calling
https_handler = urllib.request.HTTPSHandler(context=ssl.create_default_context(cafile="cacert.crt"))
the verification fails: ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129). As this post explains, certifi also automatically imports certfiles from windows cert store. Now I'm a little confused what this means if you want to verify SSL certificates in a docker container. It seems like there are two options:
- Just install the
ca-certificatespackage. It should provide the necessary public keys for most CAs. - Install your own (possibly self-signed) certificate: copy it into your docker container and tell the ca-certificates package about it by calling
update-ca-certificates. (You could also install it in windows global certificate store and it should work with docker out of the box according to this issue on github)
Unfortunately, the first approach does not seem to work for me. It raises the same Verification Error as above. Even worse, since I don't know which .crt file is used to verify certificates without docker, the second option is not a possibility either. Here's the Dockerfile:
# start with a python env that already has urllib3 installed
FROM company_harbor/base_py3_container
ENV HTTP_PROXY="my_company_proxy"
ENV HTTPS_PROXY="my_company_proxy"
# install ca certificates
RUN apt-get update && \
apt-get install ca-certificates -y && \
apt-get clean
RUN pip install --upgrade certifi --trusted-host=pypi.org --trusted-host=files.pythonhosted.org
# what I would do if I found the right .crt file
# COPY cacert.crt /usr/share/ca-certificates/cacert.crt
# RUN chmod 644 /usr/share/ca-certificates/cacert.crt
RUN update-ca-certificates
COPY ./download_penguin.py ./download_penguin.py
CMD [ "python", "download_penguin.py" ]
What do you need to do in order to verify SSL certificates with python in docker?
CodePudding user response：
Turns out company proxies can swap SSL certificates in a Man-in-the-middle manner.
The standard certificates from apt-get install ca-certificates or python's certifi package are not going to include these company certificates. Additionally, this is not specifically a Docker related question but a question of "How to install a root certificate on Linux". Debian to be more precise, because thats what Docker containers run by default.
This was not as straight-forward as expected. Here's what worked in the end:
Use the company's certificates in .pem format to begin with.
Rename them so they end with .crt. Do NOT use any openssl .pem to .crt transformation. In my case, every .crt file I found online was encoded in a way that made it unreadable for Notepad , Vim and alike. .pem files on the other hand looked fine.
Copy the renamed certificates to the proper ca-certificate location on your OS.
Install the certificates via
update-ca-certificates.
Translated into a Dockerfile, here's the important part:
COPY root.pem /usr/local/share/ca-certificates/root.crt
COPY proxy.pem /usr/local/share/ca-certificates/proxy.crt
RUN update-ca-certificates