SSH Key Management

SSH Key Management

Once you have multiple servers and multiple key pairs, managing SSH access becomes its own operational task. The SSH client config file (~/.ssh/config) is the single most underused tool for SSH productivity — it eliminates the need to remember IP addresses, usernames, and key paths for every server. Proper key management also means rotating keys periodically and knowing how to revoke a compromised key without locking yourself out.

Managing multiple keys with ~/.ssh/config

nano ~/.ssh/config

~/.ssh/config — example for a sysadmin managing multiple servers

# Default settings for all hosts
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    AddKeysToAgent yes

# Production web server
Host web-prod
    HostName 203.0.113.10
    User irfan
    IdentityFile ~/.ssh/prod_ed25519
    Port 2222

# Database server (only accessible via jump host)
Host db-prod
    HostName 10.0.1.20
    User irfan
    ProxyJump web-prod
    IdentityFile ~/.ssh/prod_ed25519

# Development server
Host dev
    HostName dev.example.com
    User developer
    IdentityFile ~/.ssh/dev_ed25519

# Wildcard for all staging servers
Host *.staging.example.com
    User irfan
    IdentityFile ~/.ssh/staging_ed25519

# With this config, connect using:
ssh web-prod          # Instead of: ssh -p 2222 -i ~/.ssh/prod_ed25519 irfan@203.0.113.10
ssh db-prod           # Automatically routes through jump host
ssh dev               # Short alias

# Also works with scp, rsync:
scp file.txt web-prod:/tmp/
rsync -av data/ dev:/opt/app/data/

The SSH agent

# SSH agent holds decrypted private keys in memory
# so you enter the passphrase once, not every connection

# Start the agent (usually auto-started on desktop environments)
eval "$(ssh-agent -s)"

# Add your key to the agent
ssh-add ~/.ssh/id_ed25519    # Prompts for passphrase once
ssh-add ~/.ssh/prod_ed25519

# List keys currently in the agent
ssh-add -l

# Add with a time limit (key expires from agent after 8 hours)
ssh-add -t 28800 ~/.ssh/prod_ed25519

# Remove a key from the agent
ssh-add -d ~/.ssh/prod_ed25519

# Remove all keys from agent
ssh-add -D

Key rotation

# Key rotation: replace old keys with new ones without losing access

# Step 1: Generate a new key pair
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new -C "irfan@company.com-$(date +%Y%m)"

# Step 2: Deploy the NEW public key (ADD to authorized_keys, don't replace yet)
ssh-copy-id -i ~/.ssh/id_ed25519_new.pub user@server

# Step 3: Test login with the new key
ssh -i ~/.ssh/id_ed25519_new user@server

# Step 4: If successful, remove the old key from authorized_keys on the server
# Find the old key's comment/fingerprint:
ssh-keygen -lf ~/.ssh/id_ed25519.pub    # Shows fingerprint
# On the server, remove that line from authorized_keys:
ssh user@server 'nano ~/.ssh/authorized_keys'

# Step 5: Update ~/.ssh/config to use the new key
# Step 6: Securely delete the old private key

Revoking a compromised key

# If a private key is compromised (laptop stolen, key file leaked):
# Remove it from authorized_keys on EVERY server it was deployed to

# Find which servers have the key (if you maintain a record):
grep "compromised-key-fingerprint" ~/.ssh/known_hosts    # No, doesn't work like this

# Better approach: search authorized_keys on all servers via Ansible
# ansible all -m shell -a "grep -l 'OLD_KEY_FINGERPRINT_OR_COMMENT' ~/.ssh/authorized_keys"

# Manual removal on each server:
# 1. Show the key to be removed
ssh-keygen -lf ~/.ssh/old_key.pub

# 2. Remove that line from authorized_keys
ssh user@server 'nano ~/.ssh/authorized_keys'

# 3. Verify the key no longer works
ssh -i ~/.ssh/old_key user@server    # Should fail

Conclusion

The ~/.ssh/config file is the most impactful SSH productivity tool: it stores hostname, user, port, and key path for every server so you type ssh servername instead of a long command. Use the SSH agent to avoid entering passphrases on every connection. Rotate keys periodically (every 6-12 months for production) by adding the new key before removing the old one. For key revocation, there is no shortcut — you need to remove the compromised key from every server's authorized_keys file.