Comprehensive Guide to Managing SSH Key Authentication and Secure Access for Linux Servers

SSH keys are safer than passwords, but only if you manage them well: a lost laptop, a key copied to every server, or a password login left on can undo the benefit. This guide covers the whole life of an SSH key on Linux servers you run yourself, such as a VPS: creating keys, installing them on many servers, switching passwords off safely, and recovering when a device is lost. It applies to your own server; on shared hosting the server's SSH settings are managed for you (see section 10).

1. How key login works

A key pair is two files. The private key stays on your computer and never leaves it. The public key is added to ~/.ssh/authorized_keys for the account you log in to on the server. When you connect, your computer proves it holds the private key without sending it, so there is nothing for an attacker to intercept or guess.

The weak points are all on the management side: private keys copied between machines, keys without passphrases, old keys nobody removed, and password login left switched on as a fallback.

2. Create one key per device, plus a recovery key

Create keys on your own computer, never on the server:

ssh-keygen -t ed25519 -a 100 -C "priya-laptop-2026"

• Ed25519 is the recommended type. Use RSA (at least 3072 bits) only for old systems that can't accept Ed25519.
• Set a passphrase. It encrypts the private key on disk, so a stolen copy is useless on its own. -a 100 makes guessing the passphrase slower.
• Use a clear comment naming the device and year. It lets you find and remove the right key later.

Make a separate key for each computer rather than copying one private key around. Then make one more, a recovery key, and keep its private half offline: on an encrypted USB drive or in a password manager, protected by its own strong passphrase. Its only job is to get you back in when your usual device is gone.

Windows, macOS and PuTTY instructions are in generating SSH keys.

3. Install the public keys on your servers

The easiest way is ssh-copy-id, which appends the key and sets the right permissions:

ssh-copy-id -i ~/.ssh/id_ed25519.pub [email protected]
ssh-copy-id -i ~/.ssh/recovery_ed25519.pub [email protected]

Without ssh-copy-id (for example on Windows), add the single line from the .pub file to ~/.ssh/authorized_keys on the server, then fix permissions:

mkdir -p ~/.ssh && chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

For several servers, keep a list and loop over it:

while read -r host; do
  ssh-copy-id -i ~/.ssh/id_ed25519.pub "deploy@$host"
done < hosts.txt

Save typing with ~/.ssh/config on your computer:

Host web1
    HostName 203.0.113.10
    User deploy
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

Now ssh web1 is enough, and IdentitiesOnly yes stops the client offering every key it has, which can trigger "too many authentication failures".

4. Turn off password login safely

Only switch passwords off after you have logged in with a key in a new session and kept the old one open.

On current Ubuntu and Debian, and on AlmaLinux or Rocky Linux 9 and later, put your settings in a drop-in file, such as /etc/ssh/sshd_config.d/00-hardening.conf:

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password
PubkeyAuthentication yes

Use PermitRootLogin no once you log in as a sudo user. KbdInteractiveAuthentication replaces the old ChallengeResponseAuthentication name.

sudo sshd -t && sudo sshd -T | grep -Ei 'passwordauthentication|permitrootlogin|kbdinteractive'

sshd -t checks the syntax; sshd -T prints the settings the server will really use. For most options the first value sshd reads wins, and another drop-in file (cloud images often ship one) may set PasswordAuthentication yes. Starting your file name with 00- makes it load first. When the output shows passwordauthentication no, reload the service: sudo systemctl reload ssh on Ubuntu and Debian, sudo systemctl reload sshd on AlmaLinux and Rocky Linux.

5. Use the agent, not copies of your key

ssh-agent holds your unlocked key in memory so you type the passphrase once per session:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

On macOS, ssh-add --apple-use-keychain stores the passphrase in the Keychain. On Windows, start the built-in OpenSSH Authentication Agent service. Adding AddKeysToAgent yes to ~/.ssh/config loads the key the first time you use it.

To reach a server behind another one, use ProxyJump (ssh -J bastion web1) rather than agent forwarding. With forwarding (-A), anyone with root on the middle server can use your agent while you are connected.

6. Plan for a lost laptop or a dead disk

Keep a copy of the recovery key in two places, and write down which servers each key is on. A list of hosts, users and key comments is enough.

7. Rotate and revoke keys

List the keys a server trusts, with their fingerprints and comments:

ssh-keygen -lf ~/.ssh/authorized_keys

Remove a key by its comment, keeping a backup of the file first:

cp ~/.ssh/authorized_keys ~/.ssh/authorized_keys.bak
sed -i '/priya-laptop-2026/d' ~/.ssh/authorized_keys

Remove keys the day a device is lost or a person leaves, and review every server's list at least once a year. You can also limit a key in authorized_keys, for example from="198.51.100.0/24" in front of the key so it works only from your office network.

8. Hardware keys and SSH certificates

• Hardware security keys (FIDO2). With OpenSSH 8.2 or later, ssh-keygen -t ed25519-sk creates a key that can't be used without the physical key plugged in and touched. The private part never leaves the device.
• SSH certificates. With many servers or people, sign each user's key with a certificate authority (ssh-keygen -s ca_key -I priya -n deploy -V +52w id_ed25519.pub) and set TrustedUserCAKeys on the servers. Servers then trust the CA instead of a growing list of keys, and certificates expire by themselves.
• Configuration management. Tools such as Ansible can push and remove authorized_keys entries across a fleet so nothing is forgotten.

9. Troubleshooting

• See which key is offered and why it fails: ssh -v web1.
• Check a key's fingerprint: ssh-keygen -lf ~/.ssh/id_ed25519.pub.
• Read the server log: sudo journalctl -u ssh on Ubuntu and Debian, sudo journalctl -u sshd on AlmaLinux and Rocky Linux.
• Permission denied (publickey): the key is in the wrong user's authorized_keys, or ~/.ssh is not 700 and the file not 600, or the home folder is writable by others.
• Locked out: use the console or rescue system to fix the drop-in file; on a Domain India VPS, open a ticket.

10. Running this on Domain India

• VPS: everything in this guide applies. A Domain India VPS is self-managed with full root access, and the client area has no VPS console, so keep a recovery key and never lock yourself out. See how to access your VPS using SSH.
• Shared hosting: Jailed SSH access is available on every shared hosting plan (cPanel, DirectAdmin, Webuzo). It is off by default; ask support to enable it for your account. You log in with a key, not a password. You add your own public key in the control panel (or send it to support), but you don't manage the server's SSH settings. See enabling and accessing jailed SSH.
• Windows (Plesk) shared hosting has no SSH.

Frequently asked questions

Ready to put this into practice? Compare VPS plans, read the VPS SSH guide, or open a support ticket to have jailed SSH enabled on your shared hosting.