In creating setup scripts, I have several git repos that I clone locally. This is done through a temporarily available proxy that may or may not be available later on, so I need to create all the remote branches from the remote repo as local branches that can be switched to. I have a method to extract the names of the remote repos that I want, when get stored as

[user]$ nvVar=$(git branch -r | grep -v '\->' | grep -Ev 'master|spdk\-1\.6' | cut -d'/' -f2)

This gives me variable list that can be iterated through, containing the branches I need to bring down.

[user]$ echo "$nvVar"
lightnvm
nvme-cuse
spdk

If I were doing all this manually, I would use commands like:

[user]$ git branch --track lightnvm origin/lightnvm
Branch lightnvm set up to track remote branch lightnvm from origin.

Which works fine... But when I try to loop through the variable using shell expansion, I get a failure. (FYI, if I put quotes around $nvVar, it doesn't iterate, and just tries running the whole string and fails. I have also tried to do this with an array, which also doesn't work, as well as using a while loop using the filtered output from git branch -r)

[user]$ for i in $nvVar; do git branch --track "${i}" "origin/${i}"; done

Which is supposed to produce the following git commands:

git branch --track lightnvm origin/lightnvm
git branch --track nvme-cuse origin/nvme-cuse
git branch --track spdk origin/spdk

Which seem to be identical to the same command typed in manually.. but instead, I get these errors:

fatal: 'lightnvm' is not a valid branch name.
fatal: 'nvme-cuse' is not a valid branch name.
fatal: 'spdk' is not a valid branch name.

Which makes no sense...

OS: RHEL 7.6

Git Version: 1.8.3.1

Bash Version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)

(Edit) Apparently I have some special characters being captured that are messing up the command.

there's a " ^[[m " being appended to the captured variable... Really not sure how to get rid of that without hard-coding the commands, which I had hoped to avoid

CodePudding user response：

Figured out a solution:

echo '#!/bin/bash' > gitShell
git branch -r | grep -v '\->' | grep -Ev 'master|spdk\-1\.6' | cut -d'/' -f2 | while read remote; do
  echo "git branch --track ${remote} origin/${remote}" >> gitShell
done
cat -v gitShell | sed 's/\^\[\[\m//g' > gitShell1
if /bin/bash -ex gitShell1; then
  echo 'Git repos branched'
  rm gitShell
  rm gitShell1
fi

I simply push the output to a file, then use cat -v to force the hidden characters to get displayed as normal characters, then filter them out with sed, and run the new script.

It's cumbersome, but it works. Apparently git returns "private unicode characters" in response to remote queries.

Thanks to @Cyrus for cluing me in to the fact that I had hidden characters in the original variable.

CodePudding user response：

The git branch command is not meant for writing scripts. The problem is occurring because you have color-changing text strings embedded within the branch names. For instance, ESC [ 3 1 m branch ESC [ m spells out "switch to green, print the word branch, and stop printing in green". (The git branch command uses green by default for the current branch, which is not the interesting one here, but still emits various escape sequences for non-current-branch cases.)

You should be using git for-each-ref instead of git branch. This is what Git calls a plumbing command: one that is meant for writing scripts. Its output is meant to be easily machine-parsed and not contain any traps like color-changing escape sequences. It also obviates the need for some of the subsequent tricks, as it has %(...) directives that can be used to strip the desired number of prefixes from items.

(Alternatively, it's possible to use git branch but to disable color output, e.g., git -c color.branch=never branch. But git branch does not promise not to make arbitrary changes to its output in the future, while git for-each-ref does.)

You might also consider attacking the original problem in a different way: create a "mirror clone", but then once the clone is done, rewrite the fetch refspec and remove the mirror configuration. The difference between a regular clone and a mirror clone is, in short, that a regular clone copies all¹ the commits and none² of the branches, but a mirror clone copies all of the commits and all of the branches and all the other references as well,³ and sets remote.remote.mirror to true.

¹Well, most of the commits, depending on which ones are reachable from which refs. If some objects are hidden or only findable via reflogs, you don't normally get those—but you don't normally care either, and in fact it's often desirable, e.g., after deleting an accidentally-committed 10 GB database.

²After copying the commits, a regular fetch turns branch names into remote-tracking names (origin/master for instance). The final git checkout step creates one new branch name, or if -n is given a tag name, doesn't.

³As with the "all commits", this is a sort of polite fiction: the sender might hide certain refs, in which case you don't get those refs, and presumably don't get those commits and other objects either. On the other hand, optimizations to avoid repacking might accidentally send unneeded objects: that 10 GB database might come through even when you didn't want it. Fortunately reflogs aren't refs, so this generally shouldn't happen.