shell script how to substitute variable contains newline-CodePudding

$ ls . > tfile.txt
$ flist=`cat tfile.txt`
$ echo $flist
a.txt b.txt c.txt
$ echo ${flist#* }
a.txt b.txt c.txt

Why ${flist#* } cannot output b.txt c.txt? How to handle variables contain line breaks?

CodePudding user response：

${flist#* } expands like $flist but removes the smallest portion matched by * (asterisk space, where asterisk matches zero or more characters and space matches space character) from the beginning of it.

So, it removes characters up to and including the first space from the expansion. So in your case, it prints a.txt and then sees a space and exits, thereby not printing b.txt and c.txt.

if you put echo "${line#*.txt}", then the output will be:

b.txt c.txt

See this reference for understanding #* in bash.

CodePudding user response：

Generally speaking the shell is not a great tool to process data and modify variable. (It is a great tool to work with Unix though, keep up learning and using it.) Some dialect can address this in some ways but it is still a good thing to keep in mind that complex data processing is better done outside the shell.

Most data processing is expected to be made with external tools rather than with shell primitives.

Here there is several ways to prune the first entry of the file you just read, the most idiosyncratic would be:

omit_first_line()
{
   tail -n 2 "$1"
}

flist=$(omit_first_line tfile.txt)

See how the $(…) are used instead of backticks, use of backticks is discouraged because they are hard to pair and to quote.

There is plenty of other tools that would skip the first line in a file for you, such as sed or awk.

If you are working on files, you really should consider using find and xargs, they work nicely together as in

find . -type f -print0 | xargs -0 stat

find . -type f -print | { while read filename; do printf 'Filename %s\n' "${filename}"; done }

Generally speaking, it is easier to compose filters than processing lists. Instead of storing data in lists you would do it in a classical imperative language, store your data in files, each record on a separate line, and pipe your data between filters. If you are able to just print your data, you might not need to store in a file it at all.

Now if you have good reasons to process your list in the shell, you can use positional arguments:

gobble()
(
  set -- "$@"
  shift
  echo "$@"
)

flist='a.txt b.txt c.txt'
flist=$(gobble ${flist})
echo ${flist}