Comparing static ssh keys and ssh certificates:¶

Title:: Comparing static ssh keys and ssh certificates:
Author:: Douglas O’Leary <dkoleary@olearycomputers.com>
Description:: Comparison of traditional/static keys and ssh certificates for authentication
Date created:: 02/13/24
Date updated:: 02/13/24
Disclaimer:: Standard: Use the information that follows at your own risk. If you screw up a system, don’t blame it on me…

Overview:¶

Over the past 15-20 years, I’ve supported large UNIX environments that used ssh for the primary direct access method. I have been a huge proponent of ssh keys for that period and longer primarily because, correctly configured and used, ssh keys provide 2 factor authentication to a host. Additionally, they can be locked down to source hosts/ips, commands that they’re authorized to run, configured to forward ports and other things. Truly a swiss army knife of tools. While I’ve never been able to implement it on-prem, I am also a huge fan of disabling password authentication like cloud service providers do for their linux virtual machines

Over the past few years, I’ve been hearing about ssh certificates and that they’re better than static keys. Since I had all the security issues covered about which I was concerned, I didn’t bother to work out the new paradigm; however, it’s time that I did.

This paper will document my research, findings, and thoughts regarding the comparison of static keys and ssh certificates.

The short version: It seems to be a matter of personal choice.

Certificates offer some benefits; but, those come with a cost. Those benefits can also be acheived using static keys and a configuration management tool like puppet or ansible which, to be fair, has to be used if you’re looking to configure a certificate authn environment as well. Another weakness that both paradigms share is key inventory. I’ve heard stats that anywhere from 60% to 85% of companies can’t tell you what keys are used in their environments. Neither paradigm will solve that issue out of the box; but, both can certainly be used to build that inventory.

Static key environment:¶

Static key authentication process:¶

The process for provisioning access for static key is well known:

user creates key pairs
user or admin copies public key to remote system in the locaiton sshd_is configured to look for them.
user sets up ssh agent if/as desired.

The ssh client sends an identifier to the server indicating that public key authentication is requested. If the corresponding public key is found the server encrypts a large prime number with the public key and sends it back. If the client can decrypt it - possibly only by using the private key - idenity verification is complete and the access is granted.

Environment configuration:¶

I implement the suggestions listed in Sudo v ssh/pka; however, in brief:

Logging fingerprints of keys used to access the system via updates to syslog and sshd_config.
Disabling ssh password authentication to root.
Using forced commands to log commands remotely executed on the system.
Developing a key management/inventory system.
Moving authorized_keys files to a root controlled directory.

Location of keys:¶

AuthorizedKeysFile is a parameter that can be set in sshd_config to move the location of authorized_keys. I’ve moved them to a root controlled directory with 755 perms and a filename of authorized_keys.%u. Users need to be able to access the files to use them so group privs are set - 640 in other words.

Moving files to a root controlled directory prevents users from adding new keys, removing forced commands, and makes periodic key audits much more straight forward.

Key management and inventory:¶

The initial inventory was created by scanning all hosts for ~/.ssh/authorized_keys - a tedious process, I assure you. Key fingerprint, owner, and other data were added to a mysql database.

By process, admins add entries to the db when they’re provisioning new public keys.

Syslog, on each system, is configured to forward log entries to a syslog collector. There’s a perl script that listens to the syslog stream, collecting data on ssh connections. It updates a file with connection related info like source, target, date/time, etc.

A cron process, run daily, scans the ssh log and updates the mysql table with last-used info for each key.

Alerting and monitoring:¶

The same perl script that listens to the syslog stream will alert admins if a key is used that isn’t in the database. Monitoring is done via other scripts that check for keys and commands used to administrative accounts like root, oracle, and others.

Auditing:¶

Key auditing is accomplished by periodically reviewing the keys in the database, verifying use and removing keys that are no longer needed. This could be further automated to automatically remove the keys from the target systems.

Other scripts collect evidence for audits both from the database and from the target systems.

Certificate based environments:¶

The quesion is: can certificate based environments be configured to accomplish the same tasks? The answer is yes although it’s important to note that, while some tasks will be easier, some will be harder. All can be accomplished using automation and the same configuration management tools mentioned earlier.

Certificate based authentication process:¶

User creates key pair as normal.
User sends public key to admin for key signing.
Admin signs key, specifying identity, principal, expiration times, etc, and returns cert to user.
User saves cert in ~/.ssh named ${key}-cert.pub

When the client initiates a new connection, the cert is sent to the target system. The server validates that the cert is signed by a trusted CA, is still valid, and that the principal identifed in the cert matches the one defined for the user. The server then extracts the public key from the cert and follows the normal authentication process from there.

Certificate environment configuration:¶

Certificate environment configuration is a superset of static keys. Rephrasing: you still need to accomplish the tasks - differently in a few cases - but they still need to be done.

It also involves the creation and use of a private certificate authority assuming you’re not using digicert or something similar.

Create ssh CA:
1. Create signing key:
```
ssh-keygen -t rsa -b 4096 -f ssh_ca_key
```
2. Configure sshd on remote systems to trust ssh_ca_key by modifying sshd_config:
```
TrustedUserCAKeys /etc/ssh/ca_pub_key.pem
AuthorizedPrincipalsFile /etc/ssh/principals/%u
```
3. Copy ssh_ca_key.pub to TrustedUserCAKeys
4. Ensure any/all forced command scripts are copied where needed.
Users generate key pairs as normal. Send public keys to admin

admin signs the keys and returns resulting cert file to user:

ssh-keygen -s ../ssh_ca_key \
  -I ${user} -n ${group1},${group2},...,${groupN} \
  -V +${#h/d/w} ${public_key_file}

Admin creates AuthorizedPrincipalsFile/${user} on appropriate remote hosts containing entries defined in -n ${group1},${group2},…,${groupN} one entry per line. This restricts access to host for matching users and principals.
User copies cert into ~/.ssh, renaming it as ${key}-cert.pub, adds agent if desired, then uses ssh normally.

Tasks that are the same:¶

Logging key fingerprints.
Disabling password authentication to root.
- Location of keys:
  
  Admins still need to configure ssh to look for authorized_keys files via the AuthorizedKeysFile parameter in sshd_config. Reason being use of certs is not an all or nothing affair. In other words, if the certificate authenticaiton fails, the server will look for an authorized keys file. If that’s still in the users’ home directory, the user can add keys, modify forced commands, etc.
Key managemnt and inventory:

Using certificates doesn’t automatically create an inventory so this is still up to the admin. Use of certificates may actually make this process more difficult, a topic that’ll be discussed in auditing below.
Alerting and monitoring:

Since public keys are extracted from certificates, the process discussed above should work for tracking keys used in a certificate based environment.

Tasks that are different:¶

Configuring hosts to trust CA.
Signing public keys and returning certs.
Forced commands: certificates don’t support the uses of forced commands so these would need to be configured in sshd_config via match/forcedcomand settings.
Auditing: The big one - has it’s own section:

Auditing keys and certificates:¶

There are two main questions that need to be answered regarding audit:

What keys exist in my environment?
What can they access?

The key inventory both paradigms require helps with the first question; however, failing that, if you’re using static keys, you can go look at the systems themselves to see what keys exist on them. If there are keys in /etc/sshkeys, it is a key in your environment.

Inventories using certificates can be done if and only if you retain a copy of the certificate. If the file isn’t retained, or is deleted after retention, the existance of the cert cannot be validated from anything I’ve found. Entertainingly, this is a similar problem to Azure’s storage accounts. Storage access signature (SAS) tokens, when generated, are signed and provisioned using the access keys for the account. Once created, there is no way to list them.

Details in the certificate can be seen via:

$ ssh-keygen -L -f ~/.ssh/id_rsa-cert.pub
/Users/dkoleary/.ssh/id_rsa-cert.pub:
        Type: ssh-rsa-cert-v01@openssh.com user certificate
        Public key: RSA-CERT SHA256:2BrSfwRVFv9dgwTprTDNGsKgvWaqKJyoX8cbw48EmgQ
        Signing CA: RSA SHA256:ba1RzzxG8spBG3fqeNzHjBKl+HCVOv7epG+VA5+YMbo (using rsa-sha2-512)
        Key ID: "dkoleary"
        Serial: 0
        Valid: from 2024-02-11T11:33:00 to 2024-02-11T12:34:08
        Principals:
                all
        Critical Options: (none)
        Extensions:
                permit-X11-forwarding
                permit-agent-forwarding
                permit-port-forwarding
                permit-pty
                permit-user-rc

It should be pretty easy to loop through all the certificate files in a directory to extract valid dates, principals, and other interesting data.

That doesn’t tell you what systems can be accessed with the certificate though just as looking at a static public key won’t tell you what systems it can access. It’s a shared problem in other words.

What keys can access which systems is easier using static keys. Again, look in /etc/sshkeys. If there’s a key, it can access; otherwise not.

Using certificates, an option would be to list the principals identified via the AuthorizedPrincipalsFile parameter in sshd_config. That doesn’t tell you if the user has a public key nor if it’s valid; only that if he presents a valid key, he’ll be let in.

The inability to identify what certificates exist would argue for a short time to live on the certificate. An unknown/unidentified cert that only lasts for day is less of a concern, long term, than one that never expires.

Short TTLs have their own issues, though. All but impossible without automation; but, the automation must take into account the service principals, identity, etc. Not impossible; but, involved.

The question regarding what can be accessed by which cert will involve identifying valid certs through the retained copies, then comparing/contrasting against the defined service principals on the systems. Not exactly an easy process; but, again, not impossible.

Another potential option is PAM. Assume, for the moment, that you define all certs to have the principal ‘all’ like the one displayed above. You would still have to create a user file with ‘all’ in it in the AuthorizedPrincipalsFile directory; but, you could rely on PAM to restrict which groups have access to which systems. Don’t want middleware people accessing oracle database servers, for instance.

Potential issue summary:¶

Password authentication circumvents this whole process therefore password authentication should be turned off.
Authorized_keys files: ~/.ssh/authorized_keys files provide an alternate authentication path. Neither overrides the other. Therefore, restricting these to a root controlled directory is still needed so users don’t provision their own access outside of the certificate process.
Can’t specify forced commands in the cert; needs to be done in the sshd_config file.
Certificate files:
- There is no way to identify configured certificates from the ssh CA server without a retained copy of the cert.
- Data on certs can be listed via ssh-keygen commands if and only if the local copy of the signed certificate is retained.
- This would argue for a reduced TTL for signed certificates.
- Short TTLs require more frequent cert signing causing a burden on admins and users alike
Certificate TTL:
- A process is needed for users to renew their own certificates.
- This would entail remote access to the ssh CA system and some script to sign/renew public keys and return the certificate file.
While remote server configuration is reduced, there is still need for the admin to touch files on remote systems for this to run:
- /etc/ssh/sshd_config
- forced commands
- User files in AuthorizedPrincipalsFile.

Summary:¶

It is possible to create a stable, secure secure shell environment using either static keys or certificates. Obviously, the process differs slightly for each. The key benefit of certificates, that I see, are ability to limit access duration; a key drawback is auditability.

For emphasis, both paradigms require automation and configuraiton management tools.

This whole discussion also assumes no enterprise secrets management tools. With one of those running and configured well, the balance tips towards certificates as the secrets management tool could handle the audit and key inventory.