SSH in Depth

One Encrypted Connection, Many Jobs

SSH (Secure Shell) carries admin logins, Git pushes, scp and sftp transfers, and tunnels to databases that are not exposed to the internet. It does for remote terminals what TLS does for the web, with a different trust model: there is usually no certificate authority, so security depends on how you handle host keys and user keys. Those two kinds of key explain nearly every SSH warning and error you will meet.

What Happens When You Type ssh host

SSH-2 is three protocols stacked on one TCP connection (port 22 by default):

1. Transport layer. Both sides send a version string (SSH-2.0-OpenSSH_10.0), exchange lists of supported algorithms (KEXINIT), and run an ephemeral key exchange. The server signs the exchange with its host key, proving its identity, and both sides derive symmetric session keys. As in TLS 1.3, the key exchange is ephemeral, so traffic has forward secrecy.
2. User authentication. Only now, inside the encrypted channel, does the client prove who the user is: public key, password, or keyboard-interactive prompts such as OTP codes.
3. Connection layer. One authenticated connection carries many independent channels: your shell, a file transfer, several port forwards, and agent requests all multiplexed together.

ssh -v prints this sequence and is the first thing to run when something fails. Connecting to GitHub with an empty known_hosts file:

debug1: Connecting to github.com [20.207.73.82] port 22.
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
No ED25519 host key is known for github.com and you have requested strict checking.
Host key verification failed.

sntrup761x25519-sha512 is a hybrid post-quantum key exchange: it combines classic X25519 with a lattice-based algorithm, so recorded traffic stays safe even if one of the two is broken later. OpenSSH 10.0 prefers mlkem768x25519-sha256 and falls back to what the server supports. The last two lines are the host key check, which is the part that matters most.

Host Keys: Trust on First Use

A TLS client trusts a server because a CA vouched for its name. A plain SSH client has no such chain. The first time you connect, it shows the host key fingerprint and asks you to confirm; after that the key is stored in ~/.ssh/known_hosts and checked on every connection. This is trust on first use (TOFU): safe afterwards, blind the first time.

Verify a fingerprint out of band: from the provider's documentation, the cloud console's boot log, or ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub on the server. It is simply a SHA-256 hash of the public key blob:

import base64
import hashlib
import sys

def fingerprint(line):
    '''OpenSSH SHA256 fingerprint of a known_hosts / authorized_keys / .pub line.'''
    fields = line.split()
    # known_hosts lines start with the host name(s); .pub lines start with the key type
    if not fields[0].startswith(("ssh-", "ecdsa-", "sk-")):
        fields = fields[1:]
    key_type, blob_b64 = fields[0], fields[1]
    blob = base64.b64decode(blob_b64)
    digest = hashlib.sha256(blob).digest()
    return key_type, "SHA256:" + base64.b64encode(digest).decode().rstrip("=")

for line in open(sys.argv[1]):
    if line.strip() and not line.startswith("#"):
        print(*fingerprint(line))

ssh-keyscan -t ed25519 github.com > gh.pub
python fp.py gh.pub
ssh-keygen -lf gh.pub

ssh-ed25519 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU
256 SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU github.com (ED25519)

Both match the fingerprint GitHub publishes. ssh-keyscan itself verifies nothing, so piping it straight into known_hosts is just TOFU with extra steps.

When the stored key and the presented key differ, OpenSSH refuses to connect with WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!. Usually the server was rebuilt, but this is exactly what a man-in-the-middle looks like, so confirm the new fingerprint before ssh-keygen -R hostname removes the old entry.

For fleets, TOFU does not scale. Distribute known_hosts from configuration management, or:

• SSH host certificates. Sign each host key with an internal CA (ssh-keygen -s host_ca -h -I web01 -n web01.internal host_key.pub) and trust the CA once with a known_hosts line: @cert-authority *.internal ssh-ed25519 AAAA....
• In automation, StrictHostKeyChecking accept-new records new hosts but still rejects changed keys, which is far safer than no. The same applies to libraries: paramiko's AutoAddPolicy, common in tutorials, accepts any host key and disables server authentication just as verify=False does for HTTPS; use load_system_host_keys() with RejectPolicy().

Key-Based User Authentication

ssh-keygen -t ed25519 -C "alice@laptop"            # writes ~/.ssh/id_ed25519 and .pub
ssh-copy-id -i ~/.ssh/id_ed25519.pub alice@server  # appends to ~/.ssh/authorized_keys
ssh-keygen -t ed25519-sk                            # key held on a FIDO2 hardware token

The client signs data tied to the session with its private key, so the key never crosses the network and a captured login cannot be replayed. Protect the private key with a passphrase: a stolen unencrypted key file works from anywhere. -sk keys need a touch on the token for each login, so a copied key file alone is useless.

Each line in authorized_keys can carry restrictions, which is how you give automation a key that can do exactly one thing:

restrict,from="10.20.0.0/16",command="/usr/local/bin/backup-receive" ssh-ed25519 AAAA... backup@nas

restrict disables port forwarding, agent forwarding, X11 and PTY allocation; from= limits source addresses; command= forces a single program regardless of what the client asks to run.

For more than a handful of servers, user certificates beat copying keys around: sign each user's key with a CA for a limited time (ssh-keygen -s user_ca -I alice -n alice -V +8h id_ed25519.pub), and set TrustedUserCAKeys /etc/ssh/user_ca.pub on servers. Access then expires on its own. Treat CA and private keys as described in secrets management.

The Agent and Why Agent Forwarding Is Risky

ssh-agent holds decrypted keys in memory so you type the passphrase once; ssh-add -t 1h loads a key for one hour only.

Agent forwarding (ssh -A) lets a remote host use your agent, for example to git pull on a server with your personal key. It does not copy the key, but while you stay connected, anyone with root on that host can use the forwarded socket to authenticate as you anywhere your key opens. For multi-hop access use ProxyJump (-J) instead: it tunnels through the jump host without exposing your agent there.

A Useful ~/.ssh/config

Host bastion
    HostName bastion.example.com
    User alice
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

Host db-* app-*
    ProxyJump bastion
    User deploy

Host *
    ServerAliveInterval 30
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist 10m

For each option, the first matching value wins, so specific Host blocks go at the top and Host * at the bottom. IdentitiesOnly yes stops the client offering every key in the agent (servers disconnect after MaxAuthTries attempts, producing Too many authentication failures). ServerAliveInterval keeps idle sessions alive through NAT devices that forget quiet connections. ControlMaster reuses one connection for later sessions to the same host. ssh -G app-1 prints the effective configuration for a host.

Port Forwarding in Depth

The VPN and tunneling lesson introduced -L and -D. The detail that trips people up is where each end lives:

In -L 8080:localhost:80, localhost means the server's loopback, not yours, because the server makes that connection. The target sees traffic coming from the SSH server's address. Forwarded ports bind to loopback by default; binding them to all interfaces (-L 0.0.0.0:8080:... on the client, GatewayPorts on the server for -R) exposes the tunnel to the whole network, usually by accident.

For long-running tunnels, add -N (no command), -o ExitOnForwardFailure=yes (exit if the port is taken) and a supervisor such as systemd or autossh. When a session hangs, type Enter, ~, . to kill it from the client side.

Hardening sshd

# /etc/ssh/sshd_config.d/10-hardening.conf
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
AllowGroups ssh-users
MaxAuthTries 3
AllowAgentForwarding no
X11Forwarding no
AllowTcpForwarding no      # enable only on bastions that need it

Apply it safely:

sudo sshd -t                       # syntax check; no output means OK
sudo sshd -T | grep -Ei 'passwordauth|permitroot|allowgroups'   # effective values
sudo systemctl reload ssh          # the unit is "sshd" on RHEL-family systems

Keep your existing session open and test a new login before logging out; a typo in AllowGroups locks everyone out.

A trap that catches many people: in sshd_config, the first obtained value for a keyword wins, and many distributions put Include /etc/ssh/sshd_config.d/*.conf near the top of the main file. If a cloud image ships a drop-in that sets PasswordAuthentication yes, your no further down the main file is silently ignored. Always confirm with sshd -T rather than reading the file.

Password guessing against port 22 never stops. With passwords disabled it is mostly log noise, which fail2ban or firewall rate limits reduce (see rate limiting and brute-force protection). Moving SSH to another port cuts the noise too, but it is not a security control. ssh-audit reports weak algorithms a server still offers.

Troubleshooting

Practice

On a test VM, set up key-only login for a user in an ssh-users group, confirm with sshd -T that passwords are off, then add a restricted authorized_keys entry whose command= runs date only. Connect with that key and ask for ls: you should get the date instead. Then try a -L forward, which restrict should refuse.