Bash quoted new-lines cause heredoc to consume first new-line-CodePudding

How to reproduce

Copy following example into a terminal session;

gawk '{
  print $0;
}' <<'EOF'
first
second
third
EOF

Press <Enter>, then <Arrow-Up>

[user@host ~]$ gawk '{
  print $0;
}' <<'EOF'first
second
third
EOF

Notice the }' <<'EOF'first bit where the first new line is consumed.

Copy following example into a terminal session;

gawk '{ print $0; }' <<'EOF'
forth
fifth
sixth
EOF

Press <Enter>, then <Arrow-Up>

[user@host ~]$ gawk '{ print $0; }' <<'EOF'
forth
fifth
sixth
EOF

Notice the }' <<'EOF' bit where the first new line is not consumed.

Questions

How do I get Bash to append to history correctly?

I spent a number of hours web-searching for others with this issue, but with no luck, is this a known bug?

Device stats

bash --version

5.1.8(1)-release (x86_64-pc-linux-gnu)

echo "${TERM}"

xterm-256color

uname -ro

5.13.19-2-MANJARO GNU/Linux

Configuration files

~/.profile

export QT_QPA_PLATFORMTHEME="qt5ct"
export EDITOR=/usr/bin/nano
export GTK2_RC_FILES="$HOME/.gtkrc-2.0"
. "$HOME/.cargo/env"

export PATH="${HOME}/.npm-global/bin:${PATH}"

~/.bashrc

#
# ~/.bashrc
#

[[ $- != *i* ]] && return

colors() {
    local fgc bgc vals seq0

    printf "Color escapes are %s\n" '\e[${value};...;${value}m'
    printf "Values 30..37 are \e[33mforeground colors\e[m\n"
    printf "Values 40..47 are \e[43mbackground colors\e[m\n"
    printf "Value  1 gives a  \e[1mbold-faced look\e[m\n\n"

    # foreground colors
    for fgc in {30..37}; do
        # background colors
        for bgc in {40..47}; do
            fgc=${fgc#37} # white
            bgc=${bgc#40} # black

            vals="${fgc: $fgc;}${bgc}"
            vals=${vals%%;}

            seq0="${vals: \e[${vals}m}"
            printf "  %-9s" "${seq0:-(default)}"
            printf " ${seq0}TEXT\e[m"
            printf " \e[${vals: ${vals $vals;}}1mBOLD\e[m"
        done
        echo; echo
    done
}

[ -r /usr/share/bash-completion/bash_completion ] && . /usr/share/bash-completion/bash_completion

# Change the window title of X terminals
case ${TERM} in
    xterm*|rxvt*|Eterm*|aterm|kterm|gnome*|interix|konsole*)
        PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME%%.*}:${PWD/#$HOME/\~}\007"'
        ;;
    screen*)
        PROMPT_COMMAND='echo -ne "\033_${USER}@${HOSTNAME%%.*}:${PWD/#$HOME/\~}\033\\"'
        ;;
esac

use_color=true

# Set colorful PS1 only on colorful terminals.
# dircolors --print-database uses its own built-in database
# instead of using /etc/DIR_COLORS.  Try to use the external file
# first to take advantage of user additions.  Use internal bash
# globbing instead of external grep binary.
safe_term=${TERM//[^[:alnum:]]/?}   # sanitize TERM
match_lhs=""
[[ -f ~/.dir_colors   ]] && match_lhs="${match_lhs}$(<~/.dir_colors)"
[[ -f /etc/DIR_COLORS ]] && match_lhs="${match_lhs}$(</etc/DIR_COLORS)"
[[ -z ${match_lhs}    ]] \
    && type -P dircolors >/dev/null \
    && match_lhs=$(dircolors --print-database)
[[ $'\n'${match_lhs} == *$'\n'"TERM "${safe_term}* ]] && use_color=true

if ${use_color} ; then
    # Enable colors for ls, etc.  Prefer ~/.dir_colors #64489
    if type -P dircolors >/dev/null ; then
        if [[ -f ~/.dir_colors ]] ; then
            eval $(dircolors -b ~/.dir_colors)
        elif [[ -f /etc/DIR_COLORS ]] ; then
            eval $(dircolors -b /etc/DIR_COLORS)
        fi
    fi

    if [[ ${EUID} == 0 ]] ; then
        PS1='\[\033[01;31m\][\h\[\033[01;36m\] \W\[\033[01;31m\]]\$\[\033[00m\] '
    else
        PS1='\[\033[01;32m\][\u@\h\[\033[01;37m\] \W\[\033[01;32m\]]\$\[\033[00m\] '
    fi

    alias ls='ls --color=auto'
    alias grep='grep --colour=auto'
    alias egrep='egrep --colour=auto'
    alias fgrep='fgrep --colour=auto'
else
    if [[ ${EUID} == 0 ]] ; then
        # show root@ when we don't have colors
        PS1='\u@\h \W \$ '
    else
        PS1='\u@\h \w \$ '
    fi
fi

unset use_color safe_term match_lhs sh

alias cp="cp -i"                          # confirm before overwriting something
alias df='df -h'                          # human-readable sizes
alias free='free -m'                      # show sizes in MB
alias np='nano -w PKGBUILD'
alias more=less

xhost  local:root > /dev/null 2>&1

# Bash won't get SIGWINCH if another process is in the foreground.
# Enable checkwinsize so that bash will check the terminal size when
# it regains control.  #65623
# http://cnswww.cns.cwru.edu/~chet/bash/FAQ (E11)
shopt -s checkwinsize

shopt -s expand_aliases

# Enable history appending instead of overwriting.  #139609
shopt -s histappend

#
# # ex - archive extractor
# # usage: ex <file>
ex ()
{
  if [ -f $1 ] ; then
    case $1 in
      *.tar.bz2)   tar xjf $1   ;;
      *.tar.gz)    tar xzf $1   ;;
      *.bz2)       bunzip2 $1   ;;
      *.rar)       unrar x $1     ;;
      *.gz)        gunzip $1    ;;
      *.tar)       tar xf $1    ;;
      *.tbz2)      tar xjf $1   ;;
      *.tgz)       tar xzf $1   ;;
      *.zip)       unzip $1     ;;
      *.Z)         uncompress $1;;
      *.7z)        7z x $1      ;;
      *)           echo "'$1' cannot be extracted via ex()" ;;
    esac
  else
    echo "'$1' is not a valid file"
  fi
}


. "$HOME/.cargo/env"
##
# Disable auto-escaping of path variables via tab-completion
# shopt -u progcomp
shopt -s direxpand


##
# Set editor for <C-x> <C-e> keyboard-shortcut
export EDITOR="$(which vim)"

##
# Find globally installed NPM scripts
export PATH="${PATH}:${HOME}/.npm/bin"

if [[ -d "${HOME}/.config/bashrc.d" ]]; then
  for _p in "${HOME}/.config/bashrc.d/"*.bashrc; do
    [[ -x "${_p}" ]] && {
      source "${_p}"
    }
  done
fi

Notes

I've tested the example commands on other devices, such as Raspberry Pi (running their Debian based Raspian), and there are no issues with quoted new lines causing heredoc newlines to be consumed. So I'm reasonably confident there's something messed up with my Manjaro (KDE flavor) configurations.

I've also checked for differences with shopt options between devices, and have not found anything, eg.

diff <(ssh rpi source '$HOME/.bashrc'; shopt -p) <(shopt -p)

#> No diff output

... So suggestions on other things to compare are certainly welcomed!

Near as I can tell there be some bug when writing to history, because;

[user@host ~]$ history 2 | head -n -1
#> 512  gawk '{
#>   print $0;
#> }' <<'EOF'first
#> second
#> third
#> EOF

And when a session is closed history similar to the flowing is saved;

[user@host ~]$ history 7 | head -n -1
#> 487  gawk '{
#> 488    print $0;
#> 489  }' <<'EOF'first
#> 490  second
#> 491  third
#> 492  EOF

As recommended by @konsolebox I've reported this as a to the Bug Bash mailing list. And I'll do my best to update this OP here if any solutions are found.

CodePudding user response：

How do I get Bash to append to history correctly?

It looks like a legitimate bug in 5.1 (tested with 5.1 and 5.1.12) so the answer is you can't unless it gets fixed. Consider sending a bug report to [email protected]. The problem does not reproduce in 5.0.

P.S. If anyone's wondering how I tested it, I use Gentoo and have multiple versions of bash installed using the ebuilds I made. See https://github.com/konsolebox/overlay/tree/master/app-shells.