Sudo vs ssh/pka to access root ¶

Title:: Sudo vs ssh/pka to access root
Author:: Douglas O’Leary <dkoleary@olearycomputers.com>
Description:: Comparison of sudo and ssh/pka access methods to sensitive accounts.
Date created:: 02/2008
Date updated:: 08/2014
Disclaimer:: Standard: Use the information that follows at your own risk. If you screw up a system, don’t blame it on me…

Proposal:¶

Use secure shell (ssh) with public key authentication (pka) to directly access root and other privileged accounts on UNIX

The standard paradigm to control access to the root and other sensitive accounts on UNIX systems utilizes personal accounts for administrative personnel, passwords, and sudo. While generally accepted, this approach has significant vulnerabilities that can be circumvented via ssh using pka. Additional procedures and scripts can be put in place to further enhance the security of pka access to privileged accounts - something that’s difficult and/or impossible to do with sudo access.

Who am I?¶

I was a Russian Linguist in the United States Air Force for 12 years during which time I held elevated security clearances. The importance of communications, information, operations, and computer security was continually impressed upon me. In 1987, I took over supervision of the Small Computer Support Branch for my military unit. In that role, I wrote and implemented most of the regulations, policies, and procedures for computer security for my unit and wing. Additionally, I had significant input into the regulations of the Electronic Security Command. In my civilian career, I’ve generated security policies for two companies and conducted UNIX security assessments for numerous clients. I am also a (ISC2) Certified Information System Security Professional (CISSP), (ISACA) Ceritfied Information Systems Auditor (CISA), and (EC-Council) Certified Ethical Hacker (C|EH). Security in general and computer security in particular has been an integral part of my entire adult life.

Background:¶

There are two very valid reasons for prohibiting direct password-based access to root and other privileged accounts: authentication and non-repudiation.

Authentication: Authentication is the process through which information is obtained and verified before authorizing access to specific resources. For the purposes of this document, those resources are the root and other sensitive accounts on the UNIX systems.

There are three factors of authentication
- Something you know
- Something you have
- Something you are
The validitiy and trust in the authentication process increases with the number of factors employed.

Password based authentication, be it telnet, ssh, or web based, is single factor authentication. You know the password.

Most people that work in reasonably large companies are familiar with the secure ID tokens used for remote access to the company network. These employ 2-factor authentication as you have the token and know the PIN that enables it.

ssh/pka is also 2-factor authentication as you have the private key and know the passphrase to it.

The third factor is generally classified as biotmetrics (retinal scans, finger/voice prints, etc) and isn’t widely used in the civilian community except in special circumstances for authentication.
Non-repudiation: With respect to computer security, repudiation refers to a user’s ability to refuse responsibility for activities done with his or her account. Non-repudiation, therefore, refers to all policies, procedures, and technical constraints which would limit a user’s ability to refuse that responsibillity.

Root access model comparisons:¶

sudo access: As stated, the commonly accepted paradigm is to have the UNIX admin log into a system using his or her personal account, then execute sudo su - or something similar to gain root authority. The intent of this process is two-fold. First, to have the admin authenticate twice - once for the personal account and again for root. The second reason is to log the access to a sensitive account. Since this is strictly password based (1-factor) authentication, it suffers from a number of weaknesses:
- Passwords are crackable: On most major variants of UNIX, passwords can be up to 255 characters; however, only the first eight are sigificant. This restriction can sometimes be circumvented and doesn’t apply to Linux; but, it reduces the possible complexity of passwords to something that can be identified via brute force attacks. In 1989, I calculated that it would take 1,973 years to crack my password. Today, using a desktop PC, that would take significantly less time, possibly measured in weeks. Additionally, programs are readily available that will attempt to identify passwords via dictionary attacks. At one particular client, I was able to crack 85% of the accounts, including several admin accounts with sudo access to root, using these programs.
- Passwords are interceptable: telnet and ftp both transmit the password to the remote host in the clear. This means that anyone with a network sniffer can easily intercept the password.
- There’s only one authentication: Despite the intent, the current paradigm effectively authenticates the user only once. While there are two separate password checks, it’s the exact same password in both cases. If the intent is truly to force two separate authentications, then sudo access to root should be disabled in favor of entering the root password via a su command. This, however, raises a host of other security concerns.
- Possibly not even one authentication: During security audits, I routinely find the NOPASSWORD option in use in sudo configurations. This removes the second password check for the admin account to gain root privileges. Now, the black hat doesn’t even need to know the admin’s password - he just has to find an open terminal.
- Access removal isn’t automatic: During one security audit, I found an admin, gone from the company for at least six months, referenced in sudoers file for 33 systems.
- Post employment activity isn’t flagged: If someone were to gain access to one of the systems referenced above, he would automatically have access to the root account. Nothing would get flagged because, according to the configuration files, the access is perfectly valid.
- Weak authentication/non-repudiation: For all the reasons speicified above, the authentication process using sudo is weak. Since password based authentication is inherently weak, a user can easily and believably say “That wasn’t me” when presented with evidence of account malfeasance
ssh/pka: Through a deceptively simple mathmatical formula, ssh authenticates systems against a list of known hosts, then sets up an encrypted tunnel between the client and the server. All this happens before a password or public key authentication sequence begins which is one of the biggest reasons for using ssh over telnet and scp/sftp over ftp. With pka, passphrases are not transmitted to the authenticating host. Instead, the same process for host authentication is used to authenticate the user. Even if the black hats had a network sniffer or a remote system had a key logger program running, a user could log in with complete security as neither a password nor a passphrase are ever sent. While no security model is completely foolproof, ssh/pka eliminates many of the weaknesses of password based authentication:
- 2-factor authentication: As stated, pka users must have a private key and know the passphrase to it. Access or knowledge of only one of these items does the black hat no good. This increases the security of the authentication process and reduces the ability of users to deny responsibility for their actions.
- pka is not prone to password/passphrase cracking attackes: Effectively, passphrases are the passwords to the private key. Passphrases can be up to 255 characters and are significant the entire length. While I doubt my passphrase would take upwards of 2000 years to crack, I’m confident the length of time to do so is back in the realm of years instead of weeks.
- ssh/pka configuration is complex: There literally is no way to accidentally configure ssh/pka access to a system. There are a number of steps that must be completed accurately before ssh/pka access to a specific key is possible. This greatly reduces the ability of users to repudiate their actions. Since root passwords are generally shared across systems, an admin could “accidentally” access a system for which he has no legitimate need. There is no such possibility using ssh/pka.
- pka is secure: pka is the core technology used for digital signatures and openssl, the technology that implements secure web sites for banking, online shopping, and auctions. All of these online activities would be impossible if there were not a reasonable level of faith in the level of non-repudiation supplied by pka.
- With some minor modifications to the default configuration, ssh logs the same or more information than does sudo. For example, this entry shows me sudo’ing into root on myhost (line breaks added for readability):
```
Jan  1 13:26:55 2008 : oleary : HOST=myhost : TTY=pts/0 ; \
    PWD=/home/dkoleary/working/ssh/comparison ; USER=root ; \
    COMMAND=/usr/bin/su -
```
  That shows date and time, who I am, what target ID I accessed, and the command I used. This log entry does not show what system I accessed myhost from, however. While a detailed analysis of the log files would identify that, it would take some additional time.
  
  Conversely, the ssh/pka access logs entries:
```
Jan  1 13:30:51 myhost sshd[143]: Connection from 192.168.12.26 port 51798
Jan  1 13:30:51 myhost sshd[143]: Failed none for root from 192.168.12.26 port 51798 ssh2
Jan  1 13:30:51 myhost sshd[143]: Found matching DSA key: c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3
Jan  1 13:30:52 myhost sshd[143]: Accepted publickey for root from 192.168.12.26 port 51798 ssh2
Jan  1 13:30:52 myhost sshd[143]: Found matching DSA key: c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3
Jan  1 13:30:52 myhost sshd[143]: Connection closed by 192.168.12.26
```
  These log entries too show date, time, target account. They also show the system that I accessed myhost from which would enable quicker tracing of the access back to its source. Using the default configuration, it could be somewhat more difficult to identify the user; however, with a minor modification, that too is easily done:
```
# ssh-keygen -l -f ./dkoleary_laptop.pub
1024 c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3 ./dkoleary_laptop.pub
```
  Note that the key fingerprints match exactly.

Required/suggested modifications:¶

As discussed above, ssh/pka provides greatly improved security authentication and can log the same information as does sudo. There are five actions required to enable that level of security and one optional one.

Log key fingerprints:¶

required
sshd_config must be modified to enable SyslogFacility (normally auth or authpriv) and LogLevel must be set to VERBOSE. These settings enable syslog to log the fingerprints of the key pair used for access.

Allow pka but prohibit password authentication:¶

required:
PermitRootLogin must be set to without-password to prevent password based authentication into root and allow pka. On most installations of openssh, it defaults to yes which also enables password authentcation. Ssh password authentication suffers from the same weaknesses as any other password based authentication process.

Update syslog configuration as needed:¶

required:
Solaris systems don’t typically log auth.info events. In order to record the key fingerprints, syslog.conf must be updated appropirately.

Log remote command execution via forced commands:¶

required:

Secure shell, regardless of authentication mechanism, enables remote command execution. While this greatly enhances an admin’s abiilty to do his or her job, it’s equally important to log what commands are being executed. This is easily done through the use of forced commands which are run regardless of what the user enters on the command line and are thus perfect for logging the commands that admins run remotely. For example:

$ **ssh myhost ls -ld /tmp**
drwxrwxrwt  69 bin        bin          24576 Jan  4 08:32 /tmp
$ **ssh myhost tail -15 /var/adm/syslog/syslog.log**
Jan  4 08:30:49 myhost inetd[19731]: registrar/tcp: Connection from myhost (192.168.12.24) at Fri Jan  4 08:30:49 2008
Jan  4 08:32:19 myhost sshd[19794]: Connection from 192.168.12.24 port 62633
Jan  4 08:32:20 myhost sshd[19794]: Failed none for root from 192.168.12.24 port 62633 ssh2
Jan  4 08:32:20 myhost sshd[19794]: Found matching DSA key: c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3
Jan  4 08:32:20 myhost sshd[19794]: Accepted publickey for root from 192.168.12.24 port 62633 ssh2
Jan  4 08:32:20 myhost sshd[19794]: Found matching DSA key: c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3
**Jan  4 08:32:20 myhost syslog: ssh/pka executed ls -ld /tmp**
Jan  4 08:32:20 myhost sshd[19794]: Connection closed by 192.168.12.24
Jan  4 08:32:20 myhost sshd[19794]: Closing connection to 192.168.12.24
Jan  4 08:32:35 myhost sshd[19807]: Connection from 192.168.12.24 port 62634
Jan  4 08:32:36 myhost sshd[19807]: Failed none for root from 192.168.12.24 port 62634 ssh2
Jan  4 08:32:36 myhost sshd[19807]: Found matching DSA key: c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3
Jan  4 08:32:36 myhost sshd[19807]: Accepted publickey for root from 192.168.12.24 port 62634 ssh2
Jan  4 08:32:36 myhost sshd[19807]: Found matching DSA key: c2:dc:f6:dc:29:2a:86:24:f4:6c:c5:d0:61:90:a5:f3
*Jan  4 08:32:36 myhost syslog: ssh/pka executed tail -15 /var/adm/syslog/syslog.log*

The lines delineated with an * show the commands I ran and that they’re reflected in the system’s syslog.

This functionality is enabled by setting an option in the public key:

command="/root/bin/sshroot" ssh-dss AAAAB3NzaC1k[[ LOTS snipped ]]

Regardless of what’s entered on the command line, ssh will execute /root/bin/sshroot. That script logs the commands via the logger command and then runs the commands as requested. The script is available here

Develop key management procedures:¶

required:
Openssh keeps all authorized public keys in one file - one very long public key per line:
```
command="/root/bin/sshroot" ssh-dss AAAAB3NzaC1kc3MAAACBALKnuT
rmoeO6X1TI6gWSq5ysxx+BzufrbfZGRcZTAZgVTityx4I+vpP6U+IveSH+gqbgkSCjD
P6g1Jl64IIAYEchMBpIl1Kx6reEzJlAM9FQk/FhAt2EJQaiyhVq4R5FBf8E2WRJTBHx
KdBEMiNTae+/1VNhpFURDBCozcU0e2Z7AAAAFQC5ORw9d7+dSMNkfY38InxUYsnCWQA
AAIEAk3uSlj0421cRkdvgd8TFdlA0uGYb9UxV8/oJb8TboKeoebNldqXQxQzf+bJBAr
7OEd9fMZzsdjQYslf5kWB04Fi9mC3ZVMviQfDzG/Qy34JodaZGMtQmhpYwPV3O85KiB
79rhM3PafcwGc1NwkoJkw0yMg+Dx7jPKi8aiC3+QF8AAACAJEeK+UjoOZHdLTP4/jKd
ttpGNCPS/dieXwANT4rj7uavXRoezP+GY/JvKGw2q6UVjvCZ79nkjgBOK4LktSE2QqW
usR/6wmV67SbqqjGrOHw0NT72gxuOEV+THI45cwcLwQSiw/SBsXXPEdlXRYRPFDo0r0
kqLlzMhOQIDDukRIE= dkoleary@myhost.com
```
That’s one line. In a generic/default environment, if an admin were to leave the company, we would have to edit the authorized_keys file on each system individually, search for the correct key, and delete it. With dozens of potential target accounts and hundreds, if not thousands of systems, key management quickly becomes administratively prohibitive without a process in place.

Probably the best way to manage keys is the use of a configuration management tool such as cfengine, puppet, or chef. These tools centrally manage system configurations automatically enabling updates like this to take seconds instead of hours of tedious, detail oriented, and error prone effort otherwise requred.

On systems that aren’t managed by a CM tool, another option is to maintain public keys in a subdirectory, typically ~/.ssh/pub_keys
$ ls ~/.ssh/pub_keys
admin1_myhost.pub
dkoleary_laptop.pub
admin2.pub
Recreating the authorized_keys file is easy:
cd ~/.ssh/pub_keys
cat *.pub > ../authorized_keys
Adding or deleting access is a simple matter of copying in or deleting the appropriate key file, then running those two commands.

Set authorized keys files in root-only writable directory:¶

optional (but highly recommended):
While this nominally increases the work load on the SA, it does prevent people from providing access to the system without the appropriate approvals. It also prevents users from removing the forced commands logging scripts identified above. Additionally, this directory simplifies key management for a configuration management tool. Instead of looking through various home directories for public keys, the tool only looks in one directory on each system. I’ve typically used /etc/ssheksys as the location for these files. This task is accomplished with the following process:
1. Copy all .ssh/authorized_keys files to /etc/sshkeys
2. Change ownership on them to root:${primary_group_of_key_owner}
3. Update /etc/ssh/sshd_config:AuthorizedKeysFile as follows:
```
AuthorizedKeysFile      /etc/sshkeys/authorized_keys.%u
```
4. Restart sshd
5. In a different window verify access to the system
6. Troubleshoot as necessary

Monitor log files for ssh key fingerprints:¶

optional: Automatic log file monitoring is optional; daily log file review is a PCI requirement and should be a requirement of any mature security policy.
Set a daemon process to watch the log files for unauthorized finger prints. This can be as simple as grep’ing key finger prints against a local list.

At one client, I configured a centralized logging server on a management node. Using syslog-ng on the management node, I sent any lines with sshd in them through a filter script called ssh_logger which would compare the fingerprints against a cfengine managed file of key fingerprints. If there was was not a match or a match but an unauthorized access, the script would automatically page out to the on call SA. All of the SAs would receive an email in the morning showing who accessed their systems.

ssh/pka concerns:¶

No security model is perfect and ssh/pka does have a few items over which to be concerned.

Null passphrased private keys:¶

First and foremost, utilizing this model effectively moves some of the authentication off of the target system. Some admins are known to use null passphrased keys as their default, interactive keys. Null passphrased keys effectively remove the something you know factor of authentication. Several recent breaches were made possible via the theft of null passphrased ssh private keys. With the ease that ssh agents can be set up, there should be no legitimate reason to use null passphrased keys as the default, interactive key. Check Configuring ssh agents for details on how to configure ssh agents.

There are ways to migitate this concern; however, the cost of implementing them can increase rather dramatically and very quickly.

Null passphrased keys can be detectedy by trying to load them into an ssh agent. Null passphrased keys will load without the standard passphrase request providing a quick and easy method for testing whether a key is passphrase protected.

If this process is implemented, crafty but lazy admins may simply put a passphrase on the key, provide a copy, then remove the passphrase again.

That issue can be circumvented by setting up a management node and configuring the sshd on each host to accept ssh connections only from that management node. On that management node, you can then script a scan for private keys and attempt to add them to an ssh-agent procss as described above.

So, what happens if your management node crashes? You have just lost ssh access to your entire environment. You can get around this by clustering the management node, by configuring several management nodes, or a management network.

None of this addresses the fact that users can establish alternate identifies which can be stored anywhere the user has write access. The compensating control there is the script I identified above which identifies key fingerprints being used to access sensitive accounts. Even with that, though, one would hope that if a user is knowledgeable enough to establish alternate identifies, he’s smart enough to know how to use ssh agents and the risks in using unrestricted null passphrased keys.

This process of identifying security weaknesses and developing mitigating controls is obviously iterative. Ever time you address one concern, you’re opening up possibilities of others.

I generally suggest that the concern of null passphrased keys be addressed through education. If I see someone using a null passphrased key, I tell the individual why unrestricted null passphrased keys are a bad idea and how they can get the exact same ability through the use of ssh-agents.

All that being said, there is a place for null passphrased keys provided some conditions are met. Assume for the moment that an admin has a need to periodically copy a file from one node to another or run a program on a remote system as part of a larger process. Being a security conscious admin, he creates a simple scp script and implements it in cron. If the admin were to try to protect the key used in the script with a passphrase, he would have to add logic to the script to check on the following:

Is there a running agent. If not, how to start one?
Is there a valid key cached in the agent.
If there’s no key, ID the location of the key and load it, somehow ensuring that the passphrase isn’t stored in clear text.

The admin’s just taken what should be a very simple script and turned it into an engineering nightmare frought with potential errors and security issues.

This is where null passphrased keys come in. How does one prevent the key from being misused? While it’s possible to limit which hosts can access the key remotely, the best answer is to limit what commands the key is allowed to run via a slight modification to the script used in the forced commands section above.

Recall that adding the command option before the key forces that command or script to be run regardless of interactive shell or what’s requested on the command line:

command="/root/bin/sshroot" ssh-dss AAAAB3NzaC1k[[ LOTS snipped ]]

If we alter that command and the accompaning script:

command="/root/bin/scproot" ssh-dss AAAAB3NzaC1k[[ LOTS snipped ]]

In addition to logging command attempts, the scproot script verifies that the requested command is in a list of authorized commands, /root/bin/VALID_COMMANDS, in this case:

ls -ld
cmgetconf
/usr/bin/cat /var/adm/syslog/ignite.log
kmtune
kctune

If any command, other than those is requested, the script issues an error. If an interactive shell is attempted, the script issues an error. Regardless of success or error, the script logs the attempt to syslog.

So, our intrepid admin above implements a forced command option, limiting the command to the specific script he or she wants. If the private key is compromised, the only thing that can be done with it is to run the script which, assuming good design, should radically reduce the damage potential.

To summarize, three rules for null passphrased keys:

They should only be used for non-interactive scripts.
They should be locked down to the the commands needed.
They should never be the default keys.

Potential for wider damage:¶

A second concern is that this level of access can be dangerous for inexperienced or careless administrators. The following loop would remove the encrypted password file from every host listed in the file list-o-hosts (real case, believe it or not) resulting in a very long and very bad day:

for h in $(cat list-o-hosts)
do
   echo ${h}
   ssh -l root ${h} rm /etc/shadow
done

As in every other access decision, access should be granted based on business need, taking into account skill and ability. Nothing says that you have to allow every admin this access.

The compelling counter argument is the need for more efficient ability to manage systems. If you have two systems and a one hour outage window, you can dedicate 30 minutes to each system. If you have 200 sytems and that same one hour outage window, you have only 18 seconds to dedicate to each system. Uptime requirements are increasing forcing a decrease to the outage windows. With the ever growing number of systems and the decreasing maintenance windows, this type of access becomes a requirement rather than an option.

All that being said, there really is no sure way around inexperience or carelessness. Every time you make something idiot proof, nature develops a better idiot.

Key/access audit:¶

The last concern is common to sudo, specifically, how to identify and audit accesses. Assuming no centralized management of sudo, to answer this question, one has to visit each system to obtain the sudoers file.

In short, this is a corallary to the key management concept discussed above.

This is another area in which the use of a centralized configuration management tool would come in very handy. Rather than hitting every system directly, the answer would result from a simple examination of the CM tool configuration possibly with a scripted crc checksum of the relevant authorized_keys files showing that they match the one in the CM tool.

Failing that, the next helpful configuraiton would be the single root-only writable authorized_keys directory. While you would still have to hit every system, at least you’re only looking in one directory.

The most inefficient solution would be to hit every system individually and every home directory individually, realizing that not all home directories are under /home. Oracle, for instance, usually has home directories in different places on different systems, making any type of automation significantly more difficult.

Summary:¶

From a security viewpoint, particularly relating to authentication and non-repudiation, employing multifactor authentication trumps single factor authentication. As ssh/pka is multifactor, it is inherently superior to any form of password authentication. With some modifications and scripting, it can provide for automatic access removal and reporting of invalid access attempt, something that is difficult, if not impossible to do with the password/sudo access paradigm.

Feel free to contact me at dkoleary@olearycomputers.com with any questions, comments, or concerns.