I am trying to run the training of stylegan2-pytorch on a remote system. The remote system has gcc (9.3.0) installed on it. I'm using conda env that has the following installed (cudatoolkit=10.2, torch=1.5.0 , and ninja=1.8.2, gcc_linux-64=7.5.0). I encounter the following error:
RuntimeError: Error building extension 'fused': [1/2]
/home/envs/segmentation_base/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/TH -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/THC -isystem /home/envs/segmentation_base/include -isystem /home/envs/segmentation_base/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c 14 -c /home/code/semanticGAN_code/models/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
FAILED: fused_bias_act_kernel.cuda.o
/home/envs/segmentation_base/bin/nvcc -DTORCH_EXTENSION_NAME=fused -DTORCH_API_INCLUDE_EXTENSION_H -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/TH -isystem /home/envs/segmentation_base/lib/python3.6/site-packages/torch/include/THC -isystem /home/envs/segmentation_base/include -isystem /home/envs/segmentation_base/include/python3.6m -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_70,code=sm_70 --compiler-options '-fPIC' -std=c 14 -c /home/code/semanticGAN_code/models/op/fused_bias_act_kernel.cu -o fused_bias_act_kernel.cuda.o
In file included from /home/envs/segmentation_base/include/cuda_runtime.h:83,
from <command-line>:
/home/envs/segmentation_base/include/crt/host_config.h:138:2: error: #error -- unsupported GNU version! gcc versions later than 8 are not supported!
138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
| ^~~~~
ninja: build stopped: subcommand failed.
I would like to use the gcc of my conda env (gcc_linux-64=7.5.0) to build cuda. When I run gcc --version in my conda env, I get the system's gcc:
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
which gcc when my conda env is active returns:
usr/bin/gcc
I'd expect it to return gcc version 7.5.0 (the one installed in the environment). I understand that conda has different names for gcc, but the environment variables should point to the installed gcc.
Running echo $CC returns
/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc.
Following suggested solution here, I get the following upon activating my environment, but the same issue stand:
INFO: activate-binutils_linux-64.sh made the following environmental changes:
ADDR2LINE=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-addr2line
AR=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ar
AS=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-as
CXXFILT=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c filt
ELFEDIT=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-elfedit
GPROF=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gprof
LD_GOLD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld.gold
LD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld
NM=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-nm
OBJCOPY=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-objcopy
OBJDUMP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-objdump
RANLIB=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ranlib
READELF=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-readelf
SIZE=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-size
STRINGS=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strings
STRIP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strip
INFO: activate-gcc_linux-64.sh made the following environmental changes:
build_alias=x86_64-conda-linux-gnu
BUILD=x86_64-conda-linux-gnu
CC_FOR_BUILD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc
CC=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cc
CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
CMAKE_ARGS=-DCMAKE_LINKER=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-ld -DCMAKE_STRIP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-strip -DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM=NEVER -DCMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY -DCMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY -DCMAKE_FIND_ROOT_PATH=;/x86_64-conda-linux-gnu/sysroot -DCMAKE_INSTALL_PREFIX= -DCMAKE_INSTALL_LIBDIR=lib
CMAKE_PREFIX_PATH=:/home/envs/segmentation_base/x86_64-conda-linux-gnu/sysroot/usr
CONDA_BUILD_SYSROOT=/home/envs/segmentation_base/x86_64-conda-linux-gnu/sysroot
_CONDA_PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_x86_64_conda_linux_gnu
CPPFLAGS=-DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /include
CPP=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-cpp
DEBUG_CFLAGS=-march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
DEBUG_CPPFLAGS=-D_DEBUG -D_FORTIFY_SOURCE=2 -Og -isystem /include
GCC_AR=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-ar
GCC_NM=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-nm
GCC=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc
GCC_RANLIB=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-gcc-ranlib
host_alias=x86_64-conda-linux-gnu
HOST=x86_64-conda-linux-gnu
LDFLAGS=-Wl,-O2 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now -Wl,--disable-new-dtags -Wl,--gc-sections -Wl,-rpath,/lib -Wl,-rpath-link,/lib -L/lib
INFO: activate-gxx_linux-64.sh made the following environmental changes:
CXXFLAGS=-fvisibility-inlines-hidden -std=c 17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
CXX_FOR_BUILD=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c
CXX=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-c
DEBUG_CXXFLAGS=-fvisibility-inlines-hidden -std=c 17 -fmessage-length=0 -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-all -fno-plt -Og -g -Wall -Wextra -fvar-tracking-assignments -ffunction-sections -pipe -isystem /include -fdebug-prefix-map==/usr/local/src/conda/- -fdebug-prefix-map==/usr/local/src/conda-prefix
GXX=/home/envs/segmentation_base/bin/x86_64-conda-linux-gnu-g
How could one set gcc to conda gcc instead of system gcc? I understand that should be done automatically, when activating the environment through bash scripts in activate.d.
Most of the open issues (regarding unsupported GNU version!) either require sudo permission to adjust gcc version (which I don't have) or aren't accepted in the case of conda environments. I have yet to find a clear solution to this :/
TLDR: How to force conda to use own installed gcc version instead of host system gcc?
Edit 1: Added conda list output
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
_sysroot_linux-64_curr_repodata_hack 3 haa98f57_10
absl-py 1.0.0 pypi_0 pypi
albumentations 0.5.2 pypi_0 pypi
binutils_impl_linux-64 2.35.1 h27ae35d_9
binutils_linux-64 2.35.1 h454624a_30
blas 1.0 mkl
ca-certificates 2021.10.26 h06a4308_2
cachetools 4.2.4 pypi_0 pypi
certifi 2021.5.30 py36h06a4308_0
charset-normalizer 2.0.9 pypi_0 pypi
cudatoolkit 10.2.89 3 hcc
cycler 0.11.0 pypi_0 pypi
decorator 4.4.2 pypi_0 pypi
freetype 2.11.0 h70c0345_0
gcc_impl_linux-64 7.5.0 h7105cf2_17
gcc_linux-64 7.5.0 h8f34230_30
google-auth 2.3.3 pypi_0 pypi
google-auth-oauthlib 0.4.6 pypi_0 pypi
grpcio 1.42.0 pypi_0 pypi
gxx_impl_linux-64 7.5.0 h0a5bf11_17
gxx_linux-64 7.5.0 hffc177d_30
idna 3.3 pypi_0 pypi
imageio 2.8.0 pypi_0 pypi
imageio-ffmpeg 0.4.2 pypi_0 pypi
imgaug 0.4.0 pypi_0 pypi
importlib-metadata 4.8.2 pypi_0 pypi
intel-openmp 2021.4.0 h06a4308_3561
jpeg 9d h7f8727e_0
kernel-headers_linux-64 3.10.0 h57e8cba_10
kiwisolver 1.3.1 pypi_0 pypi
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-devel_linux-64 7.5.0 hbbeae57_17
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libpng 1.6.37 hbc83047_0
libstdcxx-devel_linux-64 7.5.0 hf0c5c8d_17
libstdcxx-ng 9.3.0 hd4cf53a_17
libtiff 4.2.0 h85742a9_0
libwebp-base 1.2.0 h27cfd23_0
lmdb 0.98 pypi_0 pypi
lz4-c 1.9.3 h295c915_1
markdown 3.3.6 pypi_0 pypi
matplotlib 3.3.4 pypi_0 pypi
mkl 2020.2 256
mkl-service 2.3.0 py36he8ac12f_0
mkl_fft 1.3.0 py36h54f3939_0
mkl_random 1.1.1 py36h0573a6f_0
ncurses 6.3 h7f8727e_2
networkx 2.5.1 pypi_0 pypi
ninja 1.8.2 pypi_0 pypi
numpy 1.19.5 pypi_0 pypi
numpy-base 1.19.2 py36hfa32c7d_0
oauthlib 3.1.1 pypi_0 pypi
olefile 0.46 py36_0
opencv-python 4.5.4.60 pypi_0 pypi
opencv-python-headless 4.5.4.60 pypi_0 pypi
openjpeg 2.4.0 h3ad879b_0
openssl 1.1.1l h7f8727e_0
pillow 8.4.0 pypi_0 pypi
pip 21.2.2 py36h06a4308_0
protobuf 3.19.1 pypi_0 pypi
pyasn1 0.4.8 pypi_0 pypi
pyasn1-modules 0.2.8 pypi_0 pypi
pyparsing 3.0.6 pypi_0 pypi
python 3.6.13 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi
pytorch 1.5.0 py3.6_cuda10.2.89_cudnn7.6.5_0 pytorch
pywavelets 1.1.1 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
readline 8.1 h27cfd23_0
requests 2.26.0 pypi_0 pypi
requests-oauthlib 1.3.0 pypi_0 pypi
rsa 4.8 pypi_0 pypi
scikit-image 0.17.2 pypi_0 pypi
scipy 1.5.0 pypi_0 pypi
setuptools 58.0.4 py36h06a4308_0
shapely 1.8.0 pypi_0 pypi
six 1.16.0 pyhd3eb1b0_0
sqlite 3.36.0 hc218d9a_0
sysroot_linux-64 2.17 h57e8cba_10
tensorboard 2.7.0 pypi_0 pypi
tensorboard-data-server 0.6.1 pypi_0 pypi
tensorboard-plugin-wit 1.8.0 pypi_0 pypi
tifffile 2020.9.3 pypi_0 pypi
tk 8.6.11 h1ccaba5_0
torchvision 0.6.0 py36_cu102 pytorch
typing-extensions 4.0.1 pypi_0 pypi
urllib3 1.26.7 pypi_0 pypi
werkzeug 2.0.2 pypi_0 pypi
wheel 0.37.0 pyhd3eb1b0_1
xz 5.2.5 h7b6447c_0
zipp 3.6.0 pypi_0 pypi
zlib 1.2.11 h7b6447c_3
zstd 1.4.9 haebb681_0
CodePudding user response：
In addition to the solution posted in this issue. I added symbolic-links that point to the conda installed gcc, which I was missing.
ln -s /home/envs/segmentation_base/bin/x86_64-conda_cos6-linux-gnu-cc gcc
ln -s /home/envs/segmentation_base/bin/x86_64-conda_cos6-linux-gnu-cpp g
CodePudding user response：
Just to share, not sure it will help you. However it shows that in standard conditions it is possible to use the conda gcc as described in the documentation instead of the system gcc.
# system gcc
which gcc && gcc --version
# /usr/bin/gcc
# gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
# creating a conda env with gcc
conda create -n gcc gcc
# activate the environment
conda activating gcc
which gcc && gcc --version
# /opt/conda/envs/gcc/bin/gcc
# gcc (GCC) 11.2.0
Here is the list of packages installed on a fresh environment created with only gcc.
# packages in environment at /opt/conda/envs/gcc:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
binutils_impl_linux-64 2.36.1 h193b22a_2 conda-forge
gcc 11.2.0 h702ea55_2 conda-forge
gcc_impl_linux-64 11.2.0 h82a94d6_11 conda-forge
kernel-headers_linux-64 2.6.32 he073ed8_15 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
libgcc-devel_linux-64 11.2.0 h0952999_11 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libsanitizer 11.2.0 he4da1e4_11 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
sysroot_linux-64 2.12 he073ed8_15 conda-forge