I am trying to use a script to loop though a rawlist and then remove some words (lines) and put it into a final list.

#! /bin/sh

while read p; do
  sed -e "s/$p//g" domains > finaldomains
done <exclude

This is what the files contain:

rawlist:
test
test2
exclude
exclude2

excludewords:
exclude
exclude2

CodePudding user response：

For performance reasons, don't use sed in a shell loop. Instead of processing the file N times, you can accomplish this task in a single pass through each file

With awk

awk '
    NR == FNR {excluded[$0]; next}
    {
        for (word in excluded)
            if ($0 ~ word)
                print
    }
' exclude domains > finaldomains

Or grep

grep -vwFf exclude domains > finaldomains

I'm assuming you want to remove lines from "domains" when an "exclude" match is made.

CodePudding user response：

...
done <exclude

exclude maybe a file that have exclude words ..? i changed file name. exclude -> excludes.txt

• you used >. so in the loop, each sed command use rawfile and overwrites previous finaldomains file. just added sed option i then it updates only one finaldomains file in loop
• even if each sed command executes correctly, there may be overlapping words like ('exclude', 'exclude2'). so i wrapped $p with <> to find exact same row

#! /bin/sh

cp domains finaldomains
while read p; do
    echo $p
    sed -i "s/\<$p\>//g" finaldomains
done < excludes.txt