I would like to add files that meet a set of conditions to a txt file for easy transfer later, I do this with:
ls -1 > AllFilesPresent.txt
value=$(<AllFilesPresent.txt)
rm AllFilesPresent.txt
for val in $value; do
case $val in
(Result*.RData) echo "$val" >> CompletedJobs.txt ;;
esac
done
I've run into the situation where some of the files are corrupted and show up as zero byte files, which i can find manually with:
find . -size 0 -maxdepth 1
How do I include adjust my loop to reject files that are zero bytes?
CodePudding user response:
The code
ls -1 > AllFilesPresent.txt
value=$(<AllFilesPresent.txt)
rm AllFilesPresent.txt
for val in $value; do
has essentially identical functionality to
for val in $(ls -1); do
That doesn't work in general. It breaks if filenames have whitespace or glob characters in them, at least. See Bash Pitfalls #1 (for f in $(ls *.mp3)). In addition, there are particular problems with using the output of ls
in programs. It's only suitable for interactive use. See Why you shouldn't parse the output of ls(1).
A correct, completely safe, and much shorter and faster, alternative is:
for val in *; do
A full solution for your question is:
shopt -s nullglob
for file in Result*.RData; do
[[ -f $file && -s $file ]] && printf '%s\n' "$file"
done >CompletedJobs.txt
shopt -s nullglob
prevents glob patterns expanding to (what amounts to ) garbage if they don't match any files.- I've replaced
val
with the more meaningful (to me anyway)file
. - The
Result*.RData
causes the loop to only process files that match that pattern. - I've added a
-f $file
test to avoid processing any non-file things (directories, fifos, ...) that might be lying around. It still allows symlinks to files through. You might not want that. You can add a! -L $file &&
at the start of the test expression if you want to rule out symlinks. - I've replaced
echo "$val"
withprintf '%s\n' "$val"
because the original code doesn't work in general. See the accepted, and excellent, answer to Why is printf better than echo?. - I've moved the redirection to
CompletedJobs.txt
outside the loop, as suggested by @CharlesDuffy in a comment. - Note that this code won't work if any of the files have newlines in their names (e.g. create one with
echo data > $'Result\n1.RData'
). That's very uncommon, but posssible. The only way to safely store general unquoted filenames in files is to separate them with ASCII NUL characters (which can't appear in file names). To do that, replace theprintf ...
withprintf '%s\0' "$file"
. That would mean thatCompletedJobs.txt
is no longer a text file though. It would also require modifications to any tools that read the file.
You could also do this just with find
:
find . -maxdepth 1 -type f -name 'Result*.RData' -not -size 0 -printf '%P\n' >CompletedJobs.txt
- The
%P
format with-printf
removes the leading./
from outputs so you getResult2.RData
instead of./Result2.RData
(whichfind
would print by default, or with the-print
option). - Replace
\n
with\0
to make the output safe for any possible filename.
CodePudding user response:
-s file True if file exists and has a size greater than zero.
ls -1 > AllFilesPresent.txt
value=$(<AllFilesPresent.txt)
rm AllFilesPresent.txt
for val in $value; do
if [[ -s "${val}" ]]; then
case $val in
(Result*.RData) echo "$val" >> CompletedJobs.txt ;;
esac
fi
done