[Remote-SSH Bug]: SSH agent forwarding broken after reconnect

[pelmers-db]

• VSCode Version: 1.93.1 (Universal)
• Local OS Version: 14.6.1 (23G93)
• Remote OS Version: Ubuntu 18
• Remote Extension/Connection Type: Server

this is basically a duplicate submission of #9662

That issue was closed as completed, however I have observed the same behavior in latest release of remote ssh extension. [10:53:17.474] Remote-SSH version: remote-ssh@0.114.3

Steps to Reproduce:

1. work normally for a while
2. go away for a while
3. re-open laptop, VS Code reconnects and most things work
4. cannot git pull/git push, since those use agent keys

Does this issue occur when you try this locally?: no

[pelmers-db]

like the previous issue, this one goes away when setting remote.SSH.useExecServer: false

[pelmers-db]

Repro'd again. On a MacOS client it's quite easy to do:

1. put the laptop to sleep
2. return, open the terminal

$ ssh-add -l                      
Error connecting to agent: No such file or directory

3. the same happens even if I open a new terminal, or close all terminals and then open more

Further investigation suggests it's something in the extension that's injecting old environment variables into the shell session.

Evidence:

1. observe output of remote - ssh extension from when I returned:

[12:27:55.639] > Found existing installation at /home/peter.elmers/.vscode-server...
[12:27:55.640] > Starting VS Code CLI...
> printenv:
>     USER=peter.elmers
>     SSH_CLIENT=130.41.208.2 47952 22
>     MOTD_SHOWN=pam
>     HOME=/home/peter.elmers
>     OLDPWD=/home/peter.elmers
>     DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
>     LOGNAME=peter.elmers
>     _=/usr/bin/sh
>     PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
>     LANG=C.UTF-8
>     SSH_AUTH_SOCK=/tmp/ssh-wnvVKJVza5/agent.73938
>     SHELL=/usr/bin/zsh
>     PWD=/home/peter.elmers
>     SSH_CONNECTION=130.41.208.2 47952 10.91.21.232 22
[12:27:55.640] > Removing old logfile at /home/peter.elmers/.vscode-server/.cli.38c31bc77e0dd6ae88a4e9cc93428cc27a56ba40.log
[12:27:55.643] > Spawned remote CLI: 73963

2. in a broken terminal:

$ env | grep SSH                  
SSH_CLIENT=130.41.208.2 54400 22
SSH_AUTH_SOCK=/tmp/ssh-fJiZOJuhV7/agent.9790
SSH_CONNECTION=130.41.208.2 54400 10.91.21.232 22

3. observe! the printed SSH_AUTH_SOCK and the remote - ssh new SSH_AUTH_SOCK do not match!
4. manually inject the new SSH_AUTH_SOCK, and it prints my keys

$ SSH_AUTH_SOCK=/tmp/ssh-wnvVKJVza5/agent.73938 ssh-add -l
4096 SHA256:blahblah peter.elmers@mylaptophostname (RSA)

[pelmers-db]

update: this can also happen when remote.SSH.useExecServer: false, which I thought fixed the problem. I believe the issue is a buggy interaction with another vs code setting, terminal.integrated.enablePersistentSessions. When that is set to true, then vs code will restore terminal state on reconnect. Apparently the state also includes incorrect SSH_AUTH_SOCK environment.

In this case the fix is to open a new terminal.

tl;dr:
remote.SSH.useExecServer: true: completely broken
remote.SSH.useExecServer: false: the restored terminal is broken, new terminals are fixed

[ehoogeveen-medweb]

FWIW, we also experience the exact same issue on Windows 11 (connecting to a Linux remote) - we have to disable remote.SSH.useExecServer to make it work at all, and still sometimes need to close the terminal with the garbage can icon and open a new one to restore functionality.

So this is not an issue specific to MacOS - perhaps it's less common on Windows because the SSH agent is disabled by default, so agent forwarding takes more steps to set up. Perhaps there are other specific requirements to make the bug happen, but I haven't found any.

[danielhass]

I stumbled into this behavior too, currently using v0.115.0 on Windows 11 with a Linux Remote.

The problem is reproducible in a consistent manner and vanishes when switching to remote.SSH.useExecServer: false with reinstall of the remote server.

As I was already involved in #9819 this looks like a similar case at least from the users perspective.

[danielhass]

Just an update from my side. I'm still able to reproduce this issue with the latest pre-release version of the Remote SSH extension at the time of writing this comment v0.116.2024111219 when using the remote.SSH.useExecServer: true setting.

[joshspicer]

This should be fixed in the latest pre-release (released earlier today 0.116.2024112219).

With exec server enabled you should see SSH_AUTH_SOCK set to a socket with a name like vscode-ssh-auth-sock-xxxxxx

josh@devbox:~$ ssh-add -l
256 SHA256:X9xxXXXXxxXxx9xx9xx9xx9xxXxXXXXx999x9XxxxxxX josh@m3.local (ED25519)
josh@devbox:~$ echo $SSH_AUTH_SOCK 
/run/user/1000/vscode-ssh-auth-sock-231840959
josh@devbox:~$ 

Please let me know if you're seeing otherwise and include your logs (+ repro steps if relevant). Thanks for folks' patience!

[ehoogeveen-medweb]

I switched to pre-release and am still having issues, but the extension version is 0.116.2024111219. 1112 instead of 1122, so I'm guessing it's an older version? There's no option to update.

Edit: Trying it in VSCode Insiders now (thanks Daniel, I hadn't considered it) and it does appear to be working!