Forem: Smallstep

DIY SSH Bastion Host

Carl Tashian — Tue, 01 Sep 2020 19:15:00 +0000

Let's build and configure a minimal SSH bastion host (jump box) from scratch, using Ubuntu 20.04 LTS.

What's a bastion host?

Bastion is a military term meaning "a projecting part of a fortification."

In the same way that a home WiFi router sits between the vast and perilous internet and the often insecure devices on a local network, a bastion host sits between the public internet and an internal network (a VPC, for example), acting as a gateway to reach the internal hosts while protecting them from direct exposure to the wilds of the public internet. Bastion hosts often run OpenSSH or a remote desktop server.

A bastion host serves as an important choke point in a network. Given its position, it can take on a lot of responsibilities: auditing and session logging, user authentication for internal hosts, and advanced threat detection. But it doesn't need to do all that. We're going to keep things simple here and build a bastion from scratch that supports the proxying of SSH connections. Then we'll talk about some fancier stuff we could do.

Why a bastion host?

If you have an internal network and you need to reach those hosts from the public internet, a bastion host is an easy option.

Do you even need a bastion? As with nearly any decision in technology, it depends. Here are some alternatives you might consider.

Alternatives to bastions

Set up an IPsec VPN

If you need deeper access to your internal network than you can get with SSH or RDP, you may want a VPN.
But IPsec VPNs add a lot of complexity and maintenance burden compared to the other options, including bastion hosts.

Set up an overlay network

An overlay network is a lighter and simpler kind of VPN that supports roaming endpoints. It still takes a bit of setup. The most common open source options for this are Wireguard and Nebula. You could run one of these at the edge of your internal network (like a VPN), or on all the hosts you want to be able to access, and tunnel SSH traffic through it. All of your clients will also need to run Wireguard or Nebula.

Use a hosted proxy

Google IAP and AWS Session Manager are hosted solutions that tunnel SSH traffic to your internal cloud network. The benefit here is that you can use cloud IAM roles for authentication and you can implement more sophisticated security policies (security key policies, or device-level policies rather than IP-based policies) that aren't feasible if you run your own bastion host. These services are free to use, but the drawback is that IAP and AWS Session Manager are more complex than pure SSH, and they add some lock-in to GCP or AWS.

Setting up a bastion

Let's make some assumptions:

We only want this bastion to forward SSH connections to our internal hosts. Because we're using SSH here, users will have to authenticate both to the bastion, then to the internal host. To complete the connection, users will need valid credentials for both hosts, but it's possible to use different credentials for the bastion and the internal host, and we're going to take advantage of that feature.
We'll have a single shared user for everyone, and no interactive terminal sessions allowed.
Users will connect to internal hosts using ssh -J [bastion] [internal host], or with the ProxyJump directive in a Match Host block of their .ssh/config.

Launch an instance.

Stand up a Linux instance on your favorite cloud provider. We'll use Ubuntu 20.04 LTS because it is simple, it's well supported, and it includes the recently-released OpenSSH 8.2.

Set up a firewall or security group policy to restrict connections to the bastion to port 22 (SSH), and, if you can, only allow connections from IPs you trust.

Configure the bastion

We'll need to do a few things to get our bastion ready.

Configure OpenSSH

We recommend enforcing Mozilla's OpenSSH security guide. Unfortunately their guide only covers up to OpenSSH 6.7. Here are the guidelines that are still relevant to OpenSSH 8.2:

Deactivate short moduli. Moduli are used for key exchange at the start of an SSH connection. Mozilla recommends only using 3071-bit or greater moduli for extra security. To enforce this, run:

awk '$5 >= 3071' /etc/ssh/moduli > /etc/ssh/moduli.tmp && mv /etc/ssh/moduli.tmp /etc/ssh/moduli`

In /etc/ssh/sshd_config, consider the following SSHD config parameters:

# Supported HostKey algorithms by order of preference.
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_rsa_key

# Password based logins are disabled - only public key based logins are allowed.
AuthenticationMethods publickey

# LogLevel VERBOSE logs user's key fingerprint on login. 
# Needed to have a clear audit track of which key was using to log in.
LogLevel VERBOSE

PermitRootLogin no

# Log sftp level file access (read/write/etc.) that would not be easily logged otherwise.
Subsystem sftp  /usr/lib/ssh/sftp-server -f AUTHPRIV -l INFO

You should also consider which algorithms and key types you'd like to support. Mozilla recommends the following key types (more restrictive than the OpenSSH defaults):

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com

On top of Mozilla's recommendations (which only cover up to OpenSSH 6.7), here are some things you can do to beef up your SSHD security:

Require a security key:

You can set up SSHD to only accept keys that use FIDO U2F security tokens.

PubkeyAcceptedKeyTypes sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com

Note: This will prevent future connections to the host using the original PEM key you got upon launch. So, you'll need to generate a new -sk type key for the ubuntu account. We wrote instructions for that here.

Note: This requires all of your clients to have OpenSSH 8.2+.

If you do not want to restrict access by IP address in your security group rules, consider some additional hardening:

Change your default SSH port. This will deter a lot of basic bots.

Port 37271

Set up port knocking

Port knocking will complicate the task of connecting to the bastion, but it could be a good option if you need your bastion to be available to any IP address.
Install intrusion detection and prevention software

Disable Forwarding

Since we're not allowing shell access, we also want to prohibit all forwarding except TCP forwarding, which ssh -J uses to support bastions.

AllowAgentForwarding no
AllowStreamLocalForwarding no
X11Forwarding no

If all you need is an SSH gateway, you can disable shell access on the bastion itself.

Match User *,!ubuntu
        ForceCommand /bin/echo 'This bastion does not support interactive commands.'

By default, SSHD's TCP port forwarding will allow the user to forward their connection to any remote TCP port in your private network. You can limit forwarding to port 22 (SSH) if you don't want other kinds of traffic to be forwarded to internal hosts:

PermitOpen *:22

Restart SSHD (and cross your fingers)

You can test your configuration with sshd -t, then restart the SSHD server. Make sure you can still ssh into the machine before you continue! 😱

Send your users SSH keys to the bastion

If you only have a few users, you can create a single account on the bastion that everyone will use, and make sure all of their public keys are added to it.
If you have lots of users, use Smallstep SSH and issue short-lived SSH certificates.

Send logs to the cloud

You can set up the AWS CloudWatch agent or the Google Cloud Logging Agent, so that your SSH logs in particular will go to the cloud. With this in place, you can set up alerts for suspicious SSH activity.

Allow emergency root access

See our SSH Emergency Access guide for a safe approach that allows emergency access to the host.

Once you have set up emergency access keys, you can disable any other option for root access—since no one will regularly have a reason to use the root account on this machine.

Set up your internal hosts to only allow SSH access from the bastion

This is an important Zero Trust policy: Any internal host you connect to should only allow SSH connections from the bastion. The easiest way to implement this is with an inbound 1G

Configure your SSH clients.

Your clients should accept new host keys offered by a known host, making host key rotation a lot easier. (This will be the default in a future OpenSSH.)

UpdateHostKeys yes

And, you can add a configuration directive to make it easier to reach your internal hosts via the bastion. Let's say all of your internal hosts have names in the .internal domain, as is the case on AWS. You can use this directive to reach them:

Host *.internal
  ProxyJump bastion.example.com

Then, just ssh host.internal to connect to an internal host via the bastion. One subtle note here: The internal hostname will be resolved via DNS lookup on the bastion, not by your local machine. So as long as the bastion knows how to look up your internal hosts by their internal names and IPs, that's all you need.

To learn more about SSH certificates visit the Smallstep website. You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!

If you’re not using SSH certificates you’re doing SSH wrong | Episode 3: An ideal SSH flow

Cass Fultz — Fri, 21 Aug 2020 19:18:44 +0000

In our previous two episodes, we covered the benefits of SSH certificates and how they can help developers, operators, and security teams better manage privileged access. Today we will look at an ideal SSH flow using certificates and your existing identity provider for single sign-on access.

An ideal SSH flow

SSH certificate authentication is the foundation of what I think is the ideal SSH flow.

To SSH, users first run a login command in their terminal (e.g., step ssh login):

$ step ssh login
Your default web browser has been opened for you to login: https://accounts.google.com/...

A browser is opened and an SSO flow is initiated at your organization's identity provider:

A web-based SSO flow makes it easy to leverage strong MFA (e.g., FIDO U2F) and any other advanced authentication capabilities your identity provider offers. Users login with a familiar flow, and removing a user from your canonical identity provider ensures prompt termination of SSH access.

Once the user completes SSO, a bearer token (e.g., an OIDC identity token) is returned to the login utility. The utility generates a new key pair and requests a signed certificate from the CA, using the bearer token to authenticate and authorize the certificate request.

The CA returns a certificate with an expiry long enough for a work day (e.g., 16-20 hours). The login utility automatically adds the signed certificate and corresponding private key to the user's ssh-agent.

Users needn't be aware of any of this detail. All they need to know is that, in order to use SSH, they must first run step ssh login. Once that's done they can use SSH like normal:

$ ssh prod01.example.com
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1036-gcp x86_64)

Last login: Wed Sep 11 04:04:51 2019 from 98.210.132.79
mmalone@prod01:~$

Like browser cookies, short-lived certificates issued by this flow are ephemeral credentials, lasting just long enough for one work day. Like logging into a website, logging into SSH creates a session. It's a simple process that must be completed, at most, once per day. This is infrequent enough that strong MFA can be used without frustrating or desensitizing users.

New private keys and certificates are generated automatically every time the user logs in, and they never touch disk. Inserting directly into ssh-agent insulates users from sensitive credentials. If a user wants to connect from a different device it's easier for them to run step ssh login there than it is to exfiltrate keys from ssh-agent and reuse them.

There are lots of possible variations of this flow. You can adjust the certificate expiry, use PAM authentication at the CA instead of SSO, generate the private key on a smart card or TPM, opt not to use ssh-agent, or move MFA to the actual SSH connection. Personally, I think this combination offers the best balance of security and usability. Indeed, relative to most existing SSH deployments it's operationally simpler, more secure, and more usable.

Critics of SSH certificate authentication say that it's new, not well supported, and the tooling doesn't exist to use certificates in practice. The truth is, certificate authentication was added in OpenSSH 5.4 almost a decade ago. It's battle tested and used in production by massive operations. And the tooling required to build this ideal SSH flow is available today.

Tools

There are lots of existing tools for managing SSH certificates. Here are a few:

step-ca is now an SSH CA (in addition to being an X.509 CA)
step makes it easy for users and hosts to get certificates from step-ca
ssh-keygen can generate root certificates and sign user & host certificates
netflix/bless is Netflix's SSH CA that runs in AWS Lambda and uses IAM
nsheridan/cashier is Intercom's SSH CA
uber/pam-ussh lets you use certificates to authorize sudo use
hashicorp/vault has an SSH secrets engine

With the appropriate configuration of step-ca you can use step to:

Get a host certificate automatically at startup

To demonstrate, let's create a new EC2 instance with the aws CLI tool. The interesting bits are tucked in some light configuration (using a user-data startup script) that gets a host certificate and enables certificate authentication for users:

$ aws ec2 run-instances --image-id ami-07d0cf3af28718ef8 \
    --count 1 --instance-type t2.micro \
    --key-name mike-test \
    --security-group-ids launch-wizard-7 \
    --user-data file://startup.sh

Note: you should be able to use our instance identity document support here, but we've got a few kinks to work out. Stay tuned.

Get a user certificate using SSO (OAuth OIDC)

Now we'll use step ssh certificate locally (you can brew install step) to generate a new key pair, get a certificate from the CA using SSO, and automatically add the certificate and private key to ssh-agent.

That sounds like a lot, but it's just one command:

$ step ssh certificate mike@smallstep.com id_ecdsa
✔ Provisioner: Google (OIDC)
Your default web browser has been opened to visit:

https://accounts.google.com/o/oauth2/v2/auth?client_id=650...

✔ CA: https://ca.internal
✔ Private Key: id_ecdsa
✔ Public Key: id_ecdsa.pub
✔ Certificate: id_ecdsa-cert.pub
✔ SSH Agent: yes

Once that's done we can SSH to the instance we just created, using certificate authentication, with no TOFU!

$ ssh mike@ec2-100-26-100-55.compute-1.amazonaws.com
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-1044-aws x86_64)

Last login: Thu Sep 12 02:25:43 2019 from 98.210.132.79
mike@ip-172-31-70-94:~$

For more info check out our getting started guide and SSH example repo. Make sure you pass the --ssh flag to step ca init when you're setting up the CA (the getting started guide doesn't do this).

There's a lot more that can be done to make SSH certificate authentication even more awesome. We're working on that. If you have any ideas, let us know!

Use SSH certificates

SSH certificate authentication does a lot to improve SSH. It eliminates spurious TOFU warnings and host key verification failures. It lets you drop complex key approval & distribution processes and extend SSO to SSH. It makes rekeying possible for hosts and easier than key reuse for users. It makes SSH keys ephemeral, making key management oversights fail-secure.

You can deploy an SSH CA and reconfigure hosts in a matter of minutes. It's easy to transition — you can continue supporting public key authentication at the same time.

SSH certificate authentication is the right way to do SSH.

To learn more about SSH certificates visit the Smallstep website. You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!

If you’re not using SSH certificates you’re doing SSH wrong | Episode 2: Certificates improve usability, operability, & security

Cass Fultz — Fri, 21 Aug 2020 19:04:52 +0000

In our previous post, we discussed the benefits of using certificates over keys for SSH. In this episode we wanted to highlight how switching to certificates improves the lives of developers, operators, and security teams.

Certificate authentication improves usability

With public key authentication, when you SSH to a remote host for the first time, you'll be presented with a security warning like this:

$ ssh ubuntu@ec2-54-161-77-102.compute-1.amazonaws.com
The authenticity of host 'ec2-54-161-77-102.compute-1.amazonaws.com (54.161.77.102)' can't be established.
ECDSA key fingerprint is SHA256:2ae53QcOB0W6HO+XtPmMXk7To/MvMuhFxTj8ZD7eSsE.
Are you sure you want to continue connecting (yes/no)?

You've probably seen this before. If you're like most people, you've been trained to ignore it by just typing "yes". That's a problem because this is a legitimate security threat. It's also a pretty horrendous user experience. I'd wager the vast majority of SSH users don't actually understand this warning.

When you SSH to a host, the host authenticates you. Your SSH client also attempts to authenticate the host. To do so your client needs to know the host's public key. Host public keys are stored in a simple database in ~/.ssh/known_hosts. If your client can't find the host's public key in this database you get this warning. It's telling you that the host can't be authenticated!

What you're supposed to do is verify the key fingerprint out-of-band by asking an administrator or consulting a database or something. But no one does that. When you type "yes" the connection proceeds without authentication and the public key is permanently added to ~/.ssh/known_hosts. This is the trust on first use (TOFU) anti-pattern.

Since certificate authentication uses certificates to communicate public key bindings, clients are always able to authenticate, even if it's the first time connecting to a host. TOFU warnings go away.

Certificate authentication also offers a convenient place to gate SSH with custom authentication: when the certificate is issued. This can be leveraged to further enhance SSH usability. In particular, it let's you extend single sign-on (SSO) to SSH. SSO for SSH is certificate authentication's biggest party trick. We'll return to this idea and see how it further enhances usability and security later. For now, let's move on to operability.

Certificate authentication improves operability

Eliminating key approval and distribution has immediate operational benefits. You're no longer wasting ops cycles on mundane key management tasks, and you eliminate any ongoing costs associated with monitoring and maintaining homegrown machinery for adding, removing, synchronizing, and auditing static public key files across your fleet.

The ability to issue SSH user certificates via a variety of authentication mechanisms also facilitates operational automation. If a cron job or script needs SSH access it can obtain an ephemeral SSH certificate automatically, when it's needed, instead of being pre-provisioned with a long-lived, static private key.

SSH public key authentication introduces some weird operational constraints around host names that certificate authentication eliminates. As we've seen, when an SSH client connects to a host for the first time it displays a TOFU warning to the user. When the user types "yes" the host's public key is added locally to ~/.ssh/known_hosts. This binding between the host name and a specific public key is permanent. If the host presents a different public key later, the user gets an even scarier host key verification failure error message that looks like this:

$ ssh ubuntu@ec2-54-161-77-102.compute-1.amazonaws.com
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:2ae53QcOB0W6HO+XtPmMXk7To/MvMuhFxTj8ZD7eSsE.
Please contact your system administrator.
Add correct host key in ~/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in ~/.ssh/known_hosts:11
ECDSA host key for ec2-54-161-77-102.compute-1.amazonaws.com has changed and you have requested strict checking.
Host key verification failed.

This makes it operationally challenging to reuse host names. If prod01.example.com has a hardware failure, and it's replaced with a new host using the same name, host key verification failures will ensue. This usually results in a bunch engineers contacting secops to tell them they're being hacked.

Ignoring host key verification failures has the exact same attack surface area as not knowing the key at all. Curiously, OpenSSH chooses to soft-fail with an easily bypassed prompt when the key isn't known (TOFU), but hard-fails with a much scarier and harder to bypass error when there's a mismatch.

In any case, certificates fix all of this since a current name-to-public-key binding is communicated when a connection is established. Changing the host's public key is fine, as long as the host also gets a new certificate. You can safely reuse host names and even run multiple hosts with the same name. You'll never see a host key verification failure again. Beyond name reuse, we'll soon see that eliminating host key verification failures is one of the many ways certificate authentication facilitates good security hygiene.

Certificate authentication improves security

While the SSH protocol itself is secure, public key authentication encourages a bunch of bad security practices and makes good security hygiene hard to achieve.

With public key authentication, keys are trusted permanently. A compromised private key or illegitimate key binding may go unnoticed or unreported for a long time. Key management oversight (e.g., forgetting to remove an ex-employee's public keys from hosts) results in SSH failing open: unauthorized access without end.

Certificates, on the other hand, expire. In an incident — a mistake, theft, misuse, or key exfiltration of any form — compromised SSH credentials will expire automatically, without intervention, even if the incident goes unnoticed or unreported. SSH certificates are fail-secure. Access expires naturally if no action is taken to extend it. And when SSH users and hosts check in periodically with your CA to renew their credentials, a complete audit record is produced as a byproduct.

We've already seen how public key authentication trains users to ignore serious security warnings (TOFU) and triggers spurious security errors. This is more than an operational nuisance. Confusion caused by host key verification failure discourages host rekeying (i.e., replacing a host's key pair). Host private keys aren't very well protected, so periodic rekeying is good practice. Rekeying may be required after a breach or after offboarding a user. But, to avoid disruption from ensuing host key verification failures, it's often not done. Certificate authentication makes rekeying hosts trivial.

Public key authentication also makes rekeying difficult for users. Key approval and distribution is annoying enough that users are reluctant to rekey, even if you've built tools to make it possible. Worse, frustrated users copy private keys and reuse them across devices, often for many years. Key reuse is a serious security sin. Private keys are never supposed to be transferred across a network. But SSH public key authentication exposes users directly to sensitive private keys, then fails to give them usable tools for key management. It's a recipe for misuse and abuse.

An SSH CA, coupled with a simple command-line client for users, can streamline key generation and insulate users from a lot of unnecessary detail. Certificate authentication can't completely eliminate all security risks, but it does facilitate SSH workflows that are more intuitive, easier to use, and harder to misuse.

To learn more about SSH certificates visit the Smallstep website. You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!

If you’re not using SSH certificates you’re doing SSH wrong | Episode 1: Keys versus Certificates

Cass Fultz — Fri, 21 Aug 2020 18:51:11 +0000

SSH is ubiquitous. It's the de-facto solution for remote administration of *nix systems. But SSH has some pretty gnarly issues when it comes to usability, operability, and security.

You're probably familiar with these issues:

SSH user experience is terrible. SSH user on-boarding is slow and manual. Connecting to new hosts produces confusing security warnings. You're left with weird new credentials to manage with little guidance on how to do so.
Operating SSH at scale is a disaster. Key approval & distribution is a silly waste of time. Host names can't be reused. Homegrown tools scatter key material across your fleet that must be cleaned up later to off-board users.
SSH encourages bad security practices. Rekeying is hard, so it's not done. Users are exposed to key material and encouraged to reuse keys across devices. Keys are trusted permanently, so mistakes are fail-open.

The good news is this is all easy to fix.

None of these issues are actually inherent to SSH. They're actually problems with SSH public key authentication. The solution is to switch to certificate authentication.

SSH certificate authentication makes SSH easier to use, easier to operate, and more secure.

Background

At smallstep, certificates are kind of our jam. We build open source software that lets you run your own private certificate authority and manage X.509 (TLS/HTTPS) certificates.

SSH certificates have been on our radar for a while. From our perspective they're all pros, no cons. But no one uses them. Why not? We've asked hundreds of people that question. Here's what we found out:

People do use SSH certificates. In fact, pretty much everyone operating at scale that knows what they're doing does (Facebook, Uber, Google, Netflix, Intercom, Lyft, etc), but...
Certificates and public key infrastructure (PKI) are hard to grok. People don't immediately understand the benefits.
There's a (small) tooling gap that exacerbates this knowledge gap. It's not hard to fill, but people are wary to do so themselves without a deeper understanding of PKI concepts.
More than anything, SSH certificates haven't gotten the press they deserve. Most people we asked hadn't heard of them at all.

We're convinced that SSH certificates are the right way to do SSH. They're not that hard to understand, and it's well worth the effort. SSH certificates deserve more press, and broader use.

Public key authentication

Most SSH deployments use public key authentication, which uses asymmetric (public key) cryptography with a public / private key pair generated for each user & host to authenticate.

The magic of asymmetric cryptography is the special correspondence between a public and private key. You can sign data with your private key and someone else can verify your signature with the corresponding public key. Like a hash, it's computationally infeasible to forge a signature. Thus, if you can verify a signature, and you know who owns the private key, you know who generated the signature.

Simple authentication can be implemented by challenging someone to sign a big random number. If I open a socket to you and send a random number, and you respond with a valid signature over that number, I must be talking to you.

This is an oversimplification, but it's more or less how SSH public key authentication works. Certificate authentication works the same way, but with an important twist that we'll get to in a moment.

To SSH to a host using public key authentication the host needs to know your public key. By default, your public key must be added to ~/.ssh/authorized_keys. Maintaining this file for every user across a fleet is operationally challenging and error prone.

SSH user onboarding with public key authentication usually starts with some baroque incantation of ssh-keygen, hopefully pulled from a runbook, but more likely cribbed from stack overflow. Next you'll be asked to submit your public key for approval and distribution. This process is typically manual and opaque. You might be asked to email an administrator or open a JIRA ticket. Then you wait. While you're doing that, some poor operator gets interrupted and told to add your key to a manifest in some repo and trigger a deploy. Once that's done you can SSH. Since key bindings are permanent, your SSH access will continue in perpetuity until someone reverses this process.

Certificate authentication

Certificate authentication eliminates key approval and distribution. Instead of scattering public keys across static files, you bind a public key to a name with a certificate. A certificate is just a data structure that includes a public key, name, and ancillary data like an expiration date and permissions. The data structure is signed by a certificate authority (CA).

$ ssh-keygen -L -f id_ecdsa-cert.pub
id_ecdsa-cert.pub:
        Type: ecdsa-sha2-nistp256-cert-v01@openssh.com user certificate
        Public key: ECDSA-CERT SHA256:O6M6oIjDm5gPm1/aTY619BgC3KSpS4c3aHVWxYh/uGQ
        Signing CA: ECDSA SHA256:EY2EXJGoPv2LA6yEbjH+sf9JjG9Rd45FH1Wt/6H1k7Y
        Key ID: "mike@example.com"
        Serial: 4309995459650363134
        Valid: from 2019-09-11T14:50:01 to 2019-09-11T18:50:01
        Principals:
                mike
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

To enable certificate authentication simply configure clients and hosts to trust any certificates signed by your CA's public key.

On each host, edit /etc/ssh/sshd_config, specifying the CA public key for verifying user certificates, the host's private key, and the host's certificate:

# Path to the CA public key for verifying user certificates
TrustedUserCAKeys /etc/ssh/ssh_user_key.pub

# Path to this host's private key and certificate
HostKey /etc/ssh/ssh_host_ecdsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub

On each client, add a line to ~/.ssh/known_hosts specifying the CA public key for verifying host certificates:

@cert-authority *.example.com ecdsa-sha2-nistp256 AAAAE...=

That's it. That's literally all that you need to do to start using certificate authentication. You can even use it alongside public key authentication to make transitioning easier.

Static keys in ~/.ssh/authorized_keys are no longer needed. Instead, peers learn one another's public keys on demand, when connections are established, by exchanging certificates. Once certificates have been exchanged the protocol proceeds as it would with public key authentication.

To learn more about SSH certificates visit the Smallstep website. You can even try our free hosted offering and experience the value of SSH certificates in under five minutes!