What works: xargs without -I {}
I find all ".txt" files in my documents folder (about 5000 found) and use wc -l on them.
find . -type f -name "*.txt" -print0 | xargs -0 wc -l >| ~/Downloads/xargs_wc.txt
Problem: xargs -I {}
I now use xargs -I {} to find the same files:
find . -type f -name "*.txt" -print0 | xargs -0 -I {} wc -l {} >| ~/Downloads/xargs_I_wc.txt
I get consistently 8 lines of error output, corresponding to 8 files missing from ~/Downloads/xargs_I_wc.txt:
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
Debugging
I do a diff between ~/Downloads/xargs_wc.txt and ~/Downloads/xargs_I_wc.txt to get the actual files.
The 8 filenames are clean, the file contents are clean, all the files are able to be wc -l'ed manually, I can open them from the Terminal, and running file on them gives, in order:
... ASCII text (wc gives 202 lines for this file)
... ASCII text (16 lines for this one)
... ASCII text (16 lines for this)
... ASCII text, with no line terminators (132 lines each for the rest)
... ASCII text, with no line terminators
... ASCII text, with no line terminators
... ASCII text, with no line terminators
... ASCII text, with no line terminators
The first 3 files are like LICENSE files, of which the first one is in its own directory, and the next two are in relatively close proximity to each other. The last 5 are data files written in TSV format, all relatively close to each other.
The only thing that I find similar with the files is that doing ls -l on them, they all have the "@" symbol as their last symbol, but that's true for a lot of the files found:
-rw-r--r--@ ...
etc ...
Debugging - doing ls instead
I use ls instead of wc:
find . -type f -name "*.txt" -print0 | xargs -0 -I {} ls -l {} >| ~/Downloads/xargs_I_ls.txt
8 similar error lines get output:
ls: {}: No such file or directory
ls: {}: No such file or directory
ls: {}: No such file or directory
ls: {}: No such file or directory
ls: {}: No such file or directory
ls: {}: No such file or directory
ls: {}: No such file or directory
ls: {}: No such file or directory
I don't know why xargs -I {} is not working.
Further debugging - manipulating offending files
I copy the directory of one of the failing files to the root of my documents folder. I do same test with xargs -I {} wc -l:
find . -type f -name "*.txt" -print0 | xargs -0 -I {} wc -l {} >| ~/Downloads/xargs_I_wc_after_copy.txt
Same 8 error lines are output:
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
I don't know what's going on, I should be getting 9 error lines.
Even further debugging
I duplicate one of the other offending files in its own directory. I run same test.
find . -type f -name "*.txt" -print0 | xargs -0 -I {} wc -l {} >| ~/Downloads/xargs_I_wc_after_duplicate.txt
This time I get 9 error lines:
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
wc: {}: open: No such file or directory
I do many manipulations on the duplicated file: I run chmod's, I empty the file, I change the file contents, I rename the file, I repeatedly save the file, but I always get 9 error lines.
I create a new blank file and copy the contents of the original offending file into it, but I don't get 10 error lines, I get 9 error lines still.
If I move an offending file one directory outside of its current directory, I get 8 error lines. If I move the file back in, I get 9 error lines again.
There are other ".txt" files in the offending files' directories, but they don't cause errors.
The directories of the offending files do not appear to be special in any way.
Apart from deleting all the offending files, or moving them away, or creating new identical-content copies in their place, I don't know how to figure out how to get rid of their errors or what the reason is for why they're erroring.
Questions
What is going on with my seemingly normal and mundane files that is causing them to fail an xargs -I {}?
How do I get more info on what is causing them to fail and find the root cause?
CodePudding user response：
I believe macOS xargs is based on a BSD version.
The FreeBSD xargs manpage states (emphasis mine):
-I replstr
Execute utility for each input line, replacing one or more occurrences of replstr in up to replacements (or 5 if no -R flag is specified) arguments to utility with the entire line of input. The resulting arguments, after replacement is done, will not be allowed to grow beyond replsize (or 255 if no -S flag is specified) bytes; this is implemented by concatenating as much of the argument containing replstr as possible, to the constructed arguments to utility, up to replsize bytes. The size limit does not apply to arguments to utility which do not contain replstr, and furthermore, no replacement will be done on utility itself.
-S replsize
Specify the amount of space (in bytes) that -I can use for replacements. The default for replsize is 255.
When paths fed in from find exceed this limit, no replacement happens.
Some of your paths exceeded the limit.
The workaround is to use the -S flag to set a higher limit.
The maximum possible path length is given by getconf PATH_MAX /. So something at least as large should be safe in this case:
find . -type f -name "*.txt" -print0 |\
xargs -0 -S 1024 -I {} ls -l {} >| ~/Downloads/xargs_I_ls.txt