Capistrano deploy suddenly stopped working (Git Permission denied)-CodePudding

I am using capistrano in order to deploy several projects to a VPS. I keep my remote repo in this case on bitbucket. This configuration has been working for years.

I have forwarding configured in my deploy.rb:

set :ssh_options, { forward_agent: true, user: fetch(:user), keys: %w(~/.ssh/SOMEKEY_rsa) }

Suddenly I am unable to deploy or deploy:check all projects from all my local machines. The only thing, that changed is, that I had to change the hostname of the VPS some time ago. And some days ago I corrected a related configuration error by correcting the new hostname entry in my VPS admin page. Could the new hostname be the cause for the error?

This is the output I receive when running a deploy:check

% cap production deploy:check                                                    
rvm 1.29.12 (latest) by Michal Papis, Piotr Kuczynski, Wayne E. Seguin [https://rvm.io]
ruby-3.0.0
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-linux]
00:00 git:wrapper
      01 mkdir -p /tmp
    ✔ 01 [email protected] 0.120s
      Uploading /tmp/git-ssh-c9068d67cac423341809.sh 100.0%
      02 chmod 700 /tmp/git-ssh-c9068d67cac423341809.sh
    ✔ 02 [email protected] 0.119s
00:00 git:check
      01 git ls-remote [email protected]:GITUSER/SOMEREPO.git HEAD
      01 [email protected]: Permission denied (publickey).
      01
      01 fatal: Could not read from remote repository.
      01
      01
      01 Please make sure you have the correct access rights
      01 and the repository exists.
#<Thread:0x00007f8a868e0568 /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing on host IP.IP.IP.IP: git exit status: 128 (SSHKit::Runner::ExecuteError)
git stdout: [email protected]: Permission denied (publickey).\r\r
fatal: Could not read from remote repository.\r
\r
Please make sure you have the correct access rights\r
and the repository exists.
git stderr: Nothing written
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/command.rb:97:in `exit_status=': git exit status: 128 (SSHKit::Command::Failed)
git stdout: [email protected]: Permission denied (publickey).\r\r
fatal: Could not read from remote repository.\r
\r
Please make sure you have the correct access rights\r
and the repository exists.
git stderr: Nothing written
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/netssh.rb:170:in `execute_command'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
    from <internal:kernel>:90:in `tap'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:80:in `execute'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/capistrano-3.17.1/lib/capistrano/scm/git.rb:83:in `git'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/capistrano-3.17.1/lib/capistrano/scm/git.rb:40:in `check_repo_is_reachable'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/capistrano-3.17.1/lib/capistrano/scm/tasks/git.rake:19:in `block (4 levels) in eval_rakefile'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:100:in `with'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/capistrano-3.17.1/lib/capistrano/scm/tasks/git.rake:18:in `block (3 levels) in eval_rakefile'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/backends/abstract.rb:31:in `run'
    from /Users/SOMEUSER/.rvm/gems/ruby-SOMEVERSION@SOMEPROJECT/gems/sshkit-1.21.2/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing on host IP.IP.IP.IP: git exit status: 128
git stdout: [email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
git stderr: Nothing written


Caused by:
SSHKit::Command::Failed: git exit status: 128
git stdout: [email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
git stderr: Nothing written

Tasks: TOP => deploy:check => git:check
(See full trace by running task with --trace)

I can push and pull to my remote git repo on bitbucket, but apparently capistrano cannot connect to it during the deployment process.

What key (from my machine, from the VPS) is missing where (on the VPS, on bitbucket)?

How can I solve this error?

UPDATE 1:

I can connect to bitbucket from my dev system, but not from the VPS.

Currently I have no keys configured on the VPS. I do not remember, if there were some and they gor deleted when I changed the hostname. I assumed capistrano used the keys from my dev machine for the whole process.

This is the output of running ssh -Tvvv [email protected] on the vps:

% ssh -Tvvv [email protected]
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f  31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "bitbucket.org" port 22
debug2: ssh_connect_direct
debug1: Connecting to bitbucket.org [SOMEIP] SOMEPORT.
debug1: Connection established.
debug1: identity file /home/SOMEUSER/.ssh/id_rsa type -1
debug1: identity file /home/SOMEUSER/.ssh/id_rsa-cert type -1
debug1: identity file /home/SOMEUSER/.ssh/id_dsa type -1
debug1: identity file /home/SOMEUSER/.ssh/id_dsa-cert type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ecdsa type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ed25519 type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ed25519-cert type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ed25519_sk type -1
debug1: identity file /home/SOMEUSER/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/SOMEUSER/.ssh/id_xmss type -1
debug1: identity file /home/SOMEUSER/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version conker_a97b73fe8d f80d5e4fabac
debug1: no match: conker_a97b73fe8d f80d5e4fabac
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to bitbucket.org:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/SOMEUSER/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/SOMEUSER/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],rsa-sha2-512,rsa-sha2-256,ssh-rsa,[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],ssh-ed25519,[email protected]
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-s
debug2: host key algorithms: ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: [email protected],hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: rsa-sha2-512
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-rsa SOMESTUFF:SOMESTUFF
debug3: hostkeys_foreach: reading file "/home/SOMEUSER/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/SOMEUSER/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from bitbucket.org
debug3: hostkeys_foreach: reading file "/home/SOMEUSER/.ssh/known_hosts"
debug3: record_hostkey: found key type RSA in file /home/SOMEUSER/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from SOMEIP
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/SOMEUSER/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_rsa 
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_dsa 
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_ecdsa 
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_ed25519 
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/SOMEUSER/.ssh/id_xmss 
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,[email protected],[email protected],ssh-rsa,rsa-sha2-512,[email protected],[email protected],rsa-sha2-256,[email protected]>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/SOMEUSER/.ssh/id_rsa
debug3: no such identity: /home/SOMEUSER/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/SOMEUSER/.ssh/id_dsa
debug3: no such identity: /home/SOMEUSER/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/SOMEUSER/.ssh/id_ecdsa
debug3: no such identity: /home/SOMEUSER/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/SOMEUSER/.ssh/id_ecdsa_sk
debug3: no such identity: /home/SOMEUSER/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/SOMEUSER/.ssh/id_ed25519
debug3: no such identity: /home/SOMEUSER/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /home/SOMEUSER/.ssh/id_ed25519_sk
debug3: no such identity: /home/SOMEUSER/.ssh/id_ed25519_sk: No such file or directory
debug1: Trying private key: /home/SOMEUSER/.ssh/id_xmss
debug3: no such identity: /home/SOMEUSER/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
[email protected]: Permission denied (publickey).

UPDATE 2:

When I try to check ssh on the vps I get:

% ssh-add -l
Could not open a connection to your authentication agent.
% ssh-add
Could not open a connection to your authentication agent.
% eval `ssh-agent -s`
Agent pid 449351
% ssh-add -l         
The agent has no identities.

UPDATE 3

Apparently on two of my machines, both run macOS Monterey 12.6.2, ssh identities are not loaded anymore after system boot.

CodePudding user response：

I was able to solve the issue for now ... at least it is currently working.

I generated new keys on the VPS and added them on bitbucket, but what finally seems to have worked is adding my local key id_rsa again to ssh. After doing so on two machines, the process is currently working on both and I can deploy again.

I do not know, why they got disabled.