Python in docker – RuntimeError: can't start new thread-CodePudding

I'm unable to debug one error myself. I'm running python 3.8.12 inside docker image on Fedora release 35 (Thirty Five) and I'm unable to spawn threads from python. It's required for boto3 transfer to run in parallel and it uses concurrent.features to do so.

The simplest example which replicates my issue without any dependencies is (copied from python docs)

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            pass

sadly output of these lines is

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 2, in <dictcomp>
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 188, in submit
    self._adjust_thread_count()
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 213, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python3.8/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

That's all I have. Is there place where should I look? I've already checked ulimit which says unlimited. I'm kind of despair where to look or what to change to debug this issue.

CodePudding user response：

Solution to this problem was to upgrade docker from version 18.06.1-ce to 20.10.7.

Why?

This is because the default seccomp profile of Docker 20.10.9 is not adjusted to support the clone() syscall wrapper of glibc 2.34 adopted in Ubuntu 21.10 and Fedora 35.

Source: ubuntu:21.10 and fedora:35 do not work on the latest Docker (20.10.9)

CodePudding user response：

I know this may not be the answer you are looking for, but threads in Python can be problematic (because of GIL). If you can, try using ProcessPoolExecutor, it also work with concurrent.futures. However, I don't know if boto3 supports it. If you invoke the boto3 from your custom code, then it should be easy to replace threads with processes and see if that is working.