Getting version clashes on requirments.txt file in docker that don't occur when I run in a loca-CodePudding

I'm getting version clashes when I try to build a dockerfile. This doesn't happen when I run pip install -r requirements.txt in a local venv. I ran pipdeptree in my local venv after installing and found no clashes. This is only occuring when I try to build with docker through the eb cli.

I'm running docker-ce 19.03.9. Both my local venv and my dockerfile are configured to run Python 3.6. I've been unable to contact the original developer and I'm not that familiar with docker so I'm not sure where to go from here other than installing different versions of docker and trying again.

As requested, here is the dockefile;

FROM       python:3.6

WORKDIR    /var/app

RUN        pip3 install virtualenv
RUN        virtualenv /var/app
RUN        /var/app/bin/pip install --upgrade pip
RUN        /var/app/bin/pip install uwsgi

RUN        useradd uwsgi -s /bin/false
RUN        mkdir /var/log/uwsgi
RUN        chown -R uwsgi:uwsgi /var/log/uwsgi

ADD        ./requirements.txt /var/app
RUN        /var/app/bin/pip install -r /var/app/requirements.txt

ADD        . /var/app

<env stuff for uwsgi>

EXPOSE     8080

ADD        uwsgi-start.sh /

CMD        []
ENTRYPOINT ["/uwsgi-start.sh"]

The requirements list is;

appnope==0.1.0
astroid==1.6.1
autopep8==1.3.4
awscli==1.16.46
backcall==0.1.0
bleach==2.1.3
boto3==1.5.34
botocore==1.12.36
Cerberus==0.9.2
certifi==2018.4.16
cffi==1.11.4
chardet==3.0.4
click==6.7
decorator==4.3.0
dnspython==1.15.0
docutils==0.14
dominate==2.3.1
ecdsa==0.13
entrypoints==0.2.3
envs==1.2.4
Eve==0.7.8
Eve-Swagger==0.0.8
Events==0.2.2
Faker==0.8.13
flake8==3.5.0
Flask==0.12.2
Flask-Bootstrap==3.3.7.1
Flask-Cors==3.0.3
flask-nav==0.6
Flask-PyMongo==0.5.1
future==0.16.0
gunicorn==19.7.1
html5lib==1.0.1
httplib2==0.10.3
idna==2.6
ipykernel==4.8.2
ipython==6.3.1
ipython-genutils==0.2.0
ipywidgets==7.2.1
isort==4.3.4
itsdangerous==0.24
jedi==0.12.0
Jinja2==2.10
jmespath==0.9.3
jsonschema==2.6.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==5.2.0
jupyter-core==4.4.0
lazy-object-proxy==1.3.1
MarkupSafe==1.1.0
mccabe==0.6.1
mistune==0.8.3
nbconvert==5.3.1
nbformat==4.4.0
notebook==5.4.1
numpy==1.14.2
oauth2client==4.1.2
pandas==0.22.0
pandocfilters==1.4.2
parso==0.2.0
pexpect==4.5.0
pickleshare==0.7.4
pkginfo==1.4.2
prompt-toolkit==1.0.15
ptyprocess==0.5.2
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycodestyle==2.3.1
pycparser==2.18
pycryptodome==3.6.1
pyflakes==1.6.0
Pygments==2.2.0
PyJWT==1.6.1
pylint==1.8.2
pymongo==3.6.0
python-dateutil==2.7.2
python-http-client==3.0.0
python-jose-cryptodome==1.3.2
pytz==2018.4
PyYAML==3.13
pyzmq==17.0.0
qtconsole==4.3.1
readme-renderer==24.0
requests==2.18.4
requests-toolbelt==0.8.0
rsa==3.4.2
s3transfer==0.1.13
Send2Trash==1.5.0
sendgrid==5.3.0
simplegeneric==0.8.1
simplejson==3.13.2
six==1.11.0
stringcase==1.2.0
terminado==0.8.1
testpath==0.3.1
text-unidecode==1.2
tornado==5.0.2
tqdm==4.28.1
traitlets==4.3.2
urllib3==1.22
visitor==0.1.3
warrant==0.6.1
watchtower==0.5.3
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==3.2.1
wrapt==1.10.11
xlrd==1.1.0
XlsxWriter==1.0.2

The first clash is between awscli and boto3;

ERROR: Cannot install -r /var/app/requirements.txt (line 4), -r /var/app/requirements.txt (line 7) and botocore==1.12.36 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested botocore==1.12.36
    awscli 1.16.46 depends on botocore==1.12.36
    boto3 1.5.34 depends on botocore<1.9.0 and >=1.8.48
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

When I remove the versioning constraints, it will then find a fault witih werkzeug.

ERROR: Cannot install -r /var/app/requirements.txt (line 22) and Werkzeug==0.14.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested Werkzeug==0.14.1
    eve 0.7.8 depends on werkzeug<=0.11.15 and >=0.9.4
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
The command '/bin/sh -c /var/app/bin/pip install -r /var/app/requirements.txt' returned a non-zero code: 1

More and more come up as I try and fix the clashes. There are a dozen or so. I cant change the dependency requirements as this could break the app, and I've not been given enough time to try and update the code-base for these changes.

CodePudding user response：

The instructions Pip helpfully links you to explain what's going on, and it is indeed a bit of a hairy situation.

There is already an Beanstalk application that's running EC2s with docker containers that work just fine with these requirements so I'm not sure why this is happening now.

As to "why this is happening now" – two things I can think of:

The Pip version in your base image has been updated; newer versions are smarter about conflicting dependencies (in that they refuse to install package constellations that might/should not work).
Because your requirements.txt isn't necessarily fully locked down; there are transitive dependencies that get installed that aren't listed in the file that have become incompatible with one another.

However, since you already do have working container images, that's great! You could simply do

$ mv requirements.txt requirements.in
$ docker run -it thatcontainerimage /var/app/bin/pip freeze -l > requirements.txt

first move your "template" requirements, as used by e.g. pip-tools (my favored tool for dependency locking) over to requirements.in, and
acquire the full list of actually installed packages from the container to a "locked" requirements.txt file.

That requirements file should hopefully then install just fine. If this still fails, the (bad) workaround is to install every requirement in a separate pip invocation; Pip will then warn about conflicting dependencies but won't refuse to install them...

As an aside, though:

Are you sure you want to explicitly specify Werkzeug==0.14.1 in your requirements when you're also using Eve==0.7.8 which explicitly doesn't claim to work with anything newer than Werkzeug 0.11`? (The same stands for e.g. MarkupSafe and Flask versions.)
Are you absolutely sure your production app image requires awscli for something? Or autopep8 and pylint? Or gunicorn, if you're running uwsgi anyway? All of those notebook-related packages? You should pare your requirements down to separate sets for e.g. development and testing, and runtime...