How to save all variables from directory, in one file-CodePudding

I would like to save all the variables that are in the directory in a separate file, cut out duplicates To begin with, I wrote all the lines with global variables in a separate file

grep -rI "\$.*" folder/ >> output.txt

Then I tried to pull out the variables of this file

cat output.txt | sed /\$.*.[{A-Z}]/p

And output was not what I expected

So how can I take needed variables, when file after grep like this:

something.text_text.txt:    - export IMAGE_NAME=${MY_REGISTY}/$MY_PR/${MY_PRNNN} something.text_text.txt:
- docker build --network host -t ${IMAGE_NAME}:${VERSION} -f $DILE_PATH --build-arg setupfile=$SET_FIL> something.text_text.txt: 
- docker push ${IMAGE_NAME}:${VERSION} something.text_text.txt:    - docker tag ${IMAGE_NAME}:${VERSION} ${IMAGE_NAME}:${MY_BUILD_REF_NAME} something.text_text.txt:    - docker push ${IMAGE_NAME}:${MY_BUILD_REF_NAME} something.text.txt:  
- /^rel_.*$/ something.text.txt:    - eval $(ssh-agent -s) something.text.txt:    - chmod 400 $MY_SSH_KEY something.text.txt:  
- ssh-add $MY_KEY something.text.txt:    - git checkout ${MY_BUIL_NAME} something.text.txt:    - git reset --hard origin/${MY_F_NAME} something.text.txt:    - mvn -s MY_settings.xml ${MTS} license:add-third-party something.text.txt:    - cat ${LICENSE_LIST_FILE} something.text.txt:      POM_XML_COMMIT_HASH_LOCAL=$(git log --oneline --follow -- pom.xml | awk '{ print $1 }' | head -n 1) || true something.text.txt:      echo POMIT_HASH_LOCAL=${PCOMMIT_HASH_LOCAL} something.text.txt:      POM_XML_COMMIT_HASH_REMOTE=$(git log --oneline origin/${MY_BUILD_REF_NAME} --follow -- pom.xml | awk '{ print $1 }' | h> something.text.txt:      echo POM_XML_COMMIT_HASH_REMOTE=${POM_OMMIT_HASH_REMOTE} something.text.txt:      if [[ ${POM_XML_COMMIT_HASH_LOCAL} = ${POMMIT_HASH_REMOTE} ]]; then something.text.txt:        echo "File pom.xml is the same for local and origin ${MY_BUILD_REF_NAME} branch." something.text.txt:        echo "New commits are presented in origin/${MY_BUILD_REF} branch for pom.xml file. Skipping." something.text.txt:    - git add -f ${LICENSE_LIST_FILE} something.text.txt:    - export MY_PUSH_URL=`echo $MY_REPOSITORY_URL | perl -pe 's#.*@(. ?(\:\d )?)/#git@\1:#'` something.text.txt:    - git remote set-url --push origin "${MY_PUSH_URL}" something.text.txt:    - git push -f -o ci.skip origin ${MY_BUILD_REF_NAME} something.text_tests.txt:    - docker login -u $MY_REGISTRY_USER -p $MY_REGISTRY_PASSWORD $MY_REGISTRY something.text_tests.txt:    - export CONFIG_FILE=${HOME}/.docker/config.json something.text_tests.txt:   
- export VERSION=$(cat current_version) something.text_tests.txt:    - export MY_PROJECT_NAME_UPPER_CASE=$(echo ${MY_PROJECT_NAME} | tr a-z A-Z) something.text_tests.txt:    - export ${MY_PROJECUPPER_CASE}_IMAGE=${MYISTRY}/${MY_PROJECT_PATH}/${MY_PROJECT_NAME}:${VERSION} something.text_tests.txt:    - docker pull ${MY_REG}/${MY_PR}/${MY_PROJEC}:${VERS}

CodePudding user response：

Try

grep -Po '\$\.*.[{_A-Z}] ' output.txt

-P makes grep using the Perl syntax
-o outputs only the matching parts

Now, improve your regex. For starters, I have already added _ to it, but it would still find ${X}{Y} (false positive) or not find ${lowercase} (false negative) and just partly find ${DIR#/} (because of extra syntax).

CodePudding user response：

You can tell grep to output only matching parts using --only-matching or simply -o. Real problem here is what really makes a valid variable identifier. This of course depends on for which shell was script written and how many different styles original author of the script used.

Let's assume something sane, identifiers starts with [a-Z] and can only contain alphanumeric characters and underscore. You can also reference same identifier using $MY_VARIABLE or ${MY_VARIABLE}.

I would go with something like that:

grep -rhIo '\$[a-zA-Z_\{\}]*' directory | sort --unique

But be aware that syntax for arrays and operations above variables will break this very quickly.

CodePudding user response：

To get correct results for ${adjacent}text maybe go for

grep -hrEo '\$\{?[A-Za-z_0-9] \}?' .

This will still not work correctly for

: <<\_
literal $text in a quoted here document
_
echo '$quoted literal text'
echo \$escaped \$dollar \$signs
etc

but for a quick and dirty attempts, maybe just ignore those corner cases, or add some sort of postprocessing to remove them. Properly solving this requires you to have a sh parser to figure out which dollar signs are quoted etc; and with eval even that won't be complete.

As a comment on other answers here, grep -P is not portable, and requires GNU grep. If you don't have that, and really require Perl regex extensions, maybe simply go with Perl.

perl -lne 'print($&) while m/\$\{?[A-Za-z_0-9] \}?/go' **/*

The **/* recursive wildcard is not portable either; if you require a POSIX-compatible script, maybe resort to

find . -type f -exec \
    'print($&) while m/\$\{?[A-Za-z_0-9] \}?/go' {}

though of course Perl isn't at all POSIX either.