I'm trying to invoke tar command via subprocess call from Python.The challenge I have is there are a lot files that get passed on to tar which is causing the command to throw the error Argument list too long: '/bin/sh'

The command I'm running is below

subprocess.call(f"ulimit -s 99999999; tar -cz -f {output_file} {file_list}", cwd=source_dir, shell=True)

To try to overcome the error, I added ulimit which doesn't seem to help. The OS I am running this on is Ubuntu-20.04 & Pyhon version is 3.8

Please could I get help to solve this problem.

CodePudding user response：

ulimit does nothing to lift the kernel constant ARG_MAX which is what you are bumping into here. In fact, the only way to increase it is typically to recompile your kernel.

If your tar supports --files-from -, use that.

subprocess.check_call(
    ['tar', '-cz', '-f', output_file, '--files-from', '-'],
    input='\n'.join(file_list), cwd=source_dir)

I obviously made assumptions about the contents of file_list (in particular, this will break if you have files whose name contains a newline character). Notice also how I avoid shell=True by passing in the command as a list of strings.