How to run Faust from Docker - ERROR: Failed building wheel for python-rocksdb-CodePudding

I'm trying to run Python Faust from Docker.

Based on this documentation: https://faust.readthedocs.io/en/latest/userguide/installation.html

I created a simple Docker file:

FROM python:3

ADD ./app/app.py /

RUN pip3 install --upgrade pip

RUN pip install -U faust

RUN pip install "faust[rocksdb]"

RUN pip install "faust[rocksdb,uvloop,fast,redis]"

CMD ["python", "./app.py"]

When I create a docker file I receive an error at the 5th stage (Step 5/7 : RUN pip install "faust[rocksdb]")

---> Running in 1e42a5e50cbe Requirement already satisfied: faust[rocksdb] in /usr/local/lib/python3.10/site-packages (1.10.4) Requirement already satisfied: terminaltables<4.0,>=3.1 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (3.1.10) Requirement already satisfied: click<8.0,>=6.7 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (7.1.2) Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.7.2) Requirement already satisfied: aiohttp-cors<2.0,>=0.7 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (0.7.0) Requirement already satisfied: mypy-extensions in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (0.4.3) Requirement already satisfied: colorclass<3.0,>=2.2 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (2.2.2) Requirement already satisfied: opentracing<2.0.0,>=1.3.0 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.3.0) Requirement already satisfied: mode<4.4,>=4.3.2 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (4.3.2) Requirement already satisfied: venusian<2.0,>=1.1 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.2.0) Requirement already satisfied: aiohttp<4.0,>=3.5.2 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (3.8.1) Requirement already satisfied: robinhood-aiokafka<1.2,>=1.1.6 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.1.6) Requirement already satisfied: croniter>=0.3.16 in /usr/local/lib/python3.10/site-packages (from faust[rocksdb]) (1.1.0) Collecting python-rocksdb>=0.6.7 Downloading python-rocksdb-0.7.0.tar.gz (219 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (1.2.0) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (21.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (1.2.0) Requirement already satisfied: charset-normalizer<3.0,>=2.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (2.0.9) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (5.2.0) Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/site-packages (from aiohttp<4.0,>=3.5.2->faust[rocksdb]) (4.0.2) Requirement already satisfied: python-dateutil in /usr/local/lib/python3.10/site-packages (from croniter>=0.3.16->faust[rocksdb]) (2.8.2) Requirement already satisfied: colorlog>=2.9.0 in /usr/local/lib/python3.10/site-packages (from mode<4.4,>=4.3.2->faust[rocksdb]) (6.6.0) Requirement already satisfied: setuptools>=25 in /usr/local/lib/python3.10/site-packages (from python-rocksdb>=0.6.7->faust[rocksdb]) (57.5.0) Requirement already satisfied: kafka-python<1.5,>=1.4.6 in /usr/local/lib/python3.10/site-packages (from robinhood-aiokafka<1.2,>=1.1.6->faust[rocksdb]) (1.4.7) Requirement already satisfied: idna>=2.0 in /usr/local/lib/python3.10/site-packages (from yarl<2.0,>=1.0->faust[rocksdb]) (3.3) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil->croniter>=0.3.16->faust[rocksdb]) (1.16.0)

And an ERROR PART:

Building wheels for collected packages: python-rocksdb Building wheel for python-rocksdb (setup.py): started ERROR: Command errored out with exit status 1: command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/setup.py'"'"'; file='"'"'/tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-9_o4ek6z cwd: /tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/ Complete output (64 lines): running bdist_wheel running build
running build_py creating build creating build/lib.linux-x86_64-3.10 creating build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/interfaces.py -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/errors.py -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/merge_operators.py -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-3.10/rocksdb
creating build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/test_memtable.py -> build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/init.py -> build/lib.linux-x86_64-3.10/rocksdb/tests copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-3.10/rocksdb/tests running egg_info writing python_rocksdb.egg-info/PKG-INFO writing dependency_links to python_rocksdb.egg-info/dependency_links.txt writing requirements to python_rocksdb.egg-info/requires.txt writing top-level names to python_rocksdb.egg-info/top_level.txt reading manifest file 'python_rocksdb.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' writing manifest file 'python_rocksdb.egg-info/SOURCES.txt' copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/rocksdb.pyx -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/backup.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/cache.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/db.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/env.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/options.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/slice.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-3.10/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-3.10/rocksdb copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-3.10/rocksdb creating build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-3.10/rocksdb/cpp running build_ext
cythoning rocksdb/_rocksdb.pyx to rocksdb/_rocksdb.cpp
/tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/.eggs/Cython-0.29.26-py3.10-linux-x86_64.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-b8y7g4hs/python-rocksdb_b1c08993fd134ac4bc59e6f5d18bcd91/rocksdb/_rocksdb.pyx tree = Parsing.p_module(s, pxd, full_module_name) building 'rocksdb._rocksdb' extension creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/rocksdb gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.10 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.10/rocksdb/_rocksdb.o -std=c 11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing -fno-rtti rocksdb/_rocksdb.cpp:705:10: fatal error: rocksdb/slice.h: No such file or directory 705 | #include "rocksdb/slice.h" | ^~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1
---------------------------------------- Building wheel for python-rocksdb (setup.py): finished with status 'error' ERROR: Failed building wheel for python-rocksdb

Can anyone help me to move on with this? I'd like to use Faust from Docker on Kubernetes.

CodePudding user response：

Read the error message, where it is clearly stated you are missing a header file:

fatal error: rocksdb/slice.h: No such file or directory 705 | #include "rocksdb/slice.h" | ^~~~~~~~~~~~~~~~~ compilation terminated. error: command '/usr/bin/gcc' failed with exit code 1

Accordingly, you'll need to build and install RocksDB. This is separate from the installation of faust[rocksdb] with pip. That simply installs python-rocksdb, the Python interface to the underlying libraries.

There is even a (third-party) RocksDB docker image based on Python 3.7 Slim.

You could use that directly or take some tricks from the Dockerfile for that image.