ssh users’ guide ¶

Title:: ssh users’ guide
Author:: Douglas O’Leary <dkoleary@olearycomputers.com>
Description:: ssh users’ guide
Date created:: 03/2008
Date updated:: 04/2009
Disclaimer:: Standard: Use the information that follows at your own risk. If you screw up a system, don’t blame it on me…

Overview:¶

Ssh is a secure alternative to telnet, ftp, rsh, and rcp. Due to inherent security vulnerabilities in the UNIX r-command suite and the fact that the ssh protocol provides a secure encrypted tunnel, the UNIX community is slowly migrating to ssh over telnet and particularly to replace the UNIX r-command suite.

Like most other protocols, the ssh protocol encompasses a server (sshd) and clients (ssh/scp). It is not a shell like the ksh/csh. It got its name from the rsh command - this is the secure alternative.

Ssh specifically handles the authentication, encryption, and integrity of data transmitted over a network. It provides capabilities supporting secure remote logins, file copies, and remote command execution. It also supports authentication agents using public/private keys that allow you to do the above functions without constantly supplying passwords or passphrases. Additionally, it will tunnel X11 displays through the encrypted connection thereby circumventing many of the inherent weaknesses in X11 (random ports, clear text data transfers, etc).

Ssh as discussed above is a protocol - a standard - that many vendors, both open-source and commercial, have implemented in different ways. There are two versions of the protocol aptly names ssh1 and ssh2. Ssh1 has a number of inherent security flaws that have finally pushed most people to implement ssh2. Most companies are using openssh which is an open source implementation of ssh2 and supports ssh1. There are pros and cons to both the commercial and open source implementations; however, the cost (free) of the open source solution is difficult to overcome. This document describes openssh; however, many of the concepts can be directly translated to version 2 compliant software installations.

O’Reilly’s SSH, The Secure Shell: The Definitive Guide (ISBN: 0-596-00011-1) truly is appropriately named. I strongly suggest getting and reading this book. Chapter 3 of the book has an excellent basic tutorial on cryptography. It’s purposely at a high level but provides sufficient background to understand what ssh is doing and to understand the ssh configuration.

Using/Configuring ssh ¶

Section Overview:¶

There are any number of ways to use ssh as it is a very complex protocol. The sections that follow start at the most basic usage and progress to the more complex. If, as you’re configuring your ssh environment, you follow the sections in order, you will have all the requisite background to understand what you’re doing at the end. Of course, if you’re already familiar with ssh, feel free to jump to whichever section you need.

Basic ssh usage ¶

This is the easiest section because you don’t do anything to configure ssh at this level - you simply use it.

If ssh/scp is not in your path, check /opt/ssh/bin, or /usr/local/bin. Once you find it, you can either add that directory to your path or create aliases for the commands.

ssh command syntax ¶

The ssh command has a very simple syntax. For the purposes of discussion, assume that I am currently on myhost1 as user oleary.

$ ssh myhost10
oleary's password:

That will open up a connection to myhost10 and log me in assuming I provide the correct password.

$ ssh -l oracle myhost10
oracle's password:

That will open a connection to myhost10 and log me in as oracle assuming, once again, that I supply the correct password.

$ ssh -l oracle myhost10 ls -l /oracle
oracle's password:

That command will do a long listing of the /oracle directory. Any information supplied after the host that isn’t an option is considered a command. Be mindful of the shell while doing this. For instance:

$ ssh myhost10 ll /tmp > ~/tmp.list
### Password prompts assumed from here out

will create a file called tmp.list in my home directory on the local machine (myhost1). If you wanted that file on myhost10, you need to enclose the command in quotes. The same thing applies for using wildcards. The shell interprets the wildcards before it gets to ssh so please be incredibly careful with commands like:

$ ssh -l root myhost10 rm *

One last example before moving on:

$ ssh -l oracle myhost10 vi /etc/tnsnames.ora

That command will actually hang. Ssh doesn’t create a psuedo tty device unless you’re logging into the system. You can, however, ask it to do so via a command line argument:

$ ssh -l oracle -t myhost10 vi /etc/tnsnames.ora

That command will connect to myhost10 and edit the tnsnames.ora file. Once you exit, you’re back on the local machine again.

scp command syntax ¶

The scp command syntax follows the rcp command syntax almost to the letter. I won’t belabor specific examples like I did for the ssh as this one is simpler. Remember that you can copy from remote to local, local to remote and remote to remote (try that one with rcp!)

The basic format is:

scp source_file_spec destination_file_spec

Files_spec can be a simple file if it’s local, can be a ${host}:${dir}/${file} if it’s remote, or can have a username associated with it ${user}@${host}:${dir}/${file} if the file belongs to a different user on the remote host.

Some examples should be self explanatory:

$ scp myhost10:.kshrc .
$ scp root@myhost10:/etc/shadow /home/bad_guy/crack_passwords/shadow
$ scp myhost10:.kshrc myhost11:.kshrc

All of the above are valid examples although one of them should be raising some eyebrows.

Public/Private key authentication process ¶

All of the configuration options that follow require the use of public and private keys. Generating and disseminating keys can be a fairly tedious process; but, once done, provides an incredible amount of flexibility and power. The entire process, which will be discussed in the following sections, is as follows:

Generating keys
Configuring the local account
Disseminating keys/updating authorized_keys file
Configuring ssh agents

Generating keys ¶

Before generating the key pairs, decide on a system that will be your primary launch point - the system from which you will usually ssh to other systems. Although not important initially, the ssh agent will use this concept. Once you’ve decided, login to that system as your normal user id. For purposes of discussion, I will assume that the default launch point is myhost1.

To generate your key pairs, execute ssh-keygen. As one of its last steps, the ssh-keygen script will ask for a passphrase. The passphrase is the key to your private key. When an authentication request arrives, the ssh compares the public and private keys. You grant access to your private key by supplying your passphrase. Once it has access to the private key, authentication continues.

Passphrases are different than passwords. UNIX passwords are a maximum of 8 characters. You can type more than that if you wish, but only the first 8 characters are considered. Passphrases, on the other hand, are significant to 255 characters. The longer the passphrase, the harder it will be to crack. Passphrases should be mixed characters, case, numbers, spaces, etc, and should be fairly easy to type. Any full sentence in either of Al Gore’s concession speaches would be a good example.

Depending on arguments, ssh-keygen will create a public/private [dr]sa key pair in your ~/.ssh directory called id_dsa and id_dsa.pub or id_rsa/id_rsa.pub. If you already have a key pair there, ssh-keygen will overwrite it. The naming convention should be pretty self explanatory - obviously the *.pub key is the public one while the other is the private. Also, the private key must be readable only by the user, no group or world read permissions. Otherwise, ssh2 will ignore it.

ssh-keygen has a number of useful command line options:

-t [dr]sa: Specifies the type of key to generate. Most commercial variants of ssh will not support rsa as there was a rather lengthy and ugly court battle over copyright infringement. openssh supports both transparently.
-f ${file}: Is the name of the identity to create: -f ego, for instance, would create a private key (ego) and a public key (ego.pub).
-c bits: The number of bits to use in the key; the greater the number, the better the encryption but the slower it.ll work. 2048 is common.
-i/-e: These options allow you to convert openssh keys to and from their ssh2 standard compliant keys. These options are very useful if you’re supporting or working w/systems that use a commercial ssh2 implementation
-l: Used to display the key fingerprint. Useful for identifying who’s accessing your systems assuming you have the logging configured correctly. See Sudo vs ssh/pka for details

An example:

$ h
myhost1
$ pwd
/home/oleary
$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oleary/.ssh/id_dsa):[return]
Created directory '/home/oleary/.ssh'.
Enter passphrase (empty for no passphrase): [passphrase]
Enter same passphrase again: [passphrase]
Your identification has been saved in /home/oleary/.ssh/id_dsa.
Your public key has been saved in /home/oleary/.ssh/id_dsa.pub.
The key fingerprint is:
f6:38:cd:a4:d9:dd:49:a1:6b:63:9c:3a:ff:06:87:e7 oleary@myhost1
In the output above, you can see the default identity file and where you enter the passphrases. To reiterate, the passphrase is the key to your ssh identity. In a later section, we will cover null passphrased keys, when and how they should be used.

Configuring the local account ¶

Public key authentication is the single biggest area where openssh differs from the ssh2 standard. Using standards compliant ssh2 implementations, you have to create an identity, which specifies which key to use by default, and update the authorizations file to provide access. With openssh, any public/private key pair is an identity with two defaults:

id_dsa/id_dsa.pub
id_rsa/id_rsa.pub

The authorization file, ~/.ssh/authorized_keys, contains the public keys to which you will grant access. On some systems, this file could be pretty long. Initially, yours will have only one line:

cd ~/.ssh
cat id_dsa.pub > authorized_keys

That’s it.

Now, from your default system, ssh to your default system (faux loop-back test). Ssh should ask for your passphrase:

$ ssh myhost1
The authenticity of host 'myhost1 (135.3.14.137)' can't be established.
RSA key fingerprint is a2:ec:75:37:71:dc:9b:c8:b2:52:7c:10:57:93:f4:6e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'myhost1,135.3.14.137' (RSA) to the list of known hosts.
Enter passphrase for key '/home/oleary/.ssh/id_dsa': [passphrase]
Last successful login: Wed Jan 21 10:20:29 EST 2009 myhost1.ndc.myco.com
Last authentication failure: Tue Jan 20 12:01:43 EST 2009 myhost3.ndc.myco.com
Last login: Wed Jan 21 10:20:29 2009 from myhost1.ndc.myco.com

If you’ve never connected to the system via ssh before, ssh will ask to add the system to the ~/.ssh/known_hosts file. Any time you connect to that host from then on, ssh will compare the host’s key against the key in your known_hosts file and alert you if there is a conflict. This prevents someone from tampering with the destination host and intercepting your traffic. Normally, you should never see any messages about the host key being different.

Once that check is done, ssh moves on to the authentication. Ssh detects the authorized keys file and checks to see if your ~/.ssh directory has a corresponding private key. In this case, it does, so it asks for the passphrase to the key. Upon receipt of a valid passphrase, ssh enables access to the system. The exact details of the process are listed in the cryptographic primer in Chapter 3 of the O’Reilly book.

Disseminating keys/updating authorized_keys file ¶

As mentioned previously, the authorized_keys file contains the public keys to which the account will grant access. In order to gain access to a remote account, then, you will need to copy your public key to the remote system and update the account’s authorized_keys file. You will need to do this for every account that you want to directly access from your launch point. For instance, UNIX admins will want to update theirs and root’s authorization file; DBAs will want to update theirs and the oracle/sap admin account’s authorization file. This will allow a user execute the command ssh -l ${usr} ${host} to gain access.

Public key filename syntax ¶

Some systems will have a lot of public keys and a long authorization file for the administrative accounts. If you’re not using a central management system, such as cfengine, for your authorized_keys files, you should consider key management at this point.

Assume you have a large authorizations file and someone on your team leaves. Common security practice is to remove his access to the systems. You do this by removing his key from the authorized_keys file and changing the account password. Here’s the problem: You have possibly 100s of systems that you have to visit individually, edit the file, and find one key out of many that all look alike to remove it. That, right there, is one of the very nice things about cfengine - you edit the file in one place and your 100s of systems pull the change within half an hour.

Another option is to keep all the public keys that are authorized to access the account as separate files similarly to the version 2 compliant installations. Regenerating the authorized_keys file is then a simple matter of:

# adding/removing the appropriate key
cat *.pub > authorized_keys

Obviously, not everyone’s public key can be named id_dsa.pub. To support a significant number of keys and to prevent file clobbering, I usually suggest a standard public key naming convention that follows these rules:

All public keys for accounts other than the local account owner will be stored under ~/.ssh/pub_keys
Public key filenames will be in the format ${owner}_${host}.pub
1. ${host} is the host where the private key counterpart to the public key is stored.
2. ${owner} is the UNIX userid of the owner of the public/private key pair.

My public key on most systems will be stored in:

~/.ssh/pub_keys/dkoleary_laptop.pub

Using this standard, recreating a corrupted authorized_keys file is as simple as:

cd ~/.ssh/pub_keys
cat *.pub > ../authorized_keys

If someone on your team leaves or moves to a different job, removing their access is equally as simple:

cd ~/.ssh/pub_keys
rm ${owner}*
cat *.pub > ../authorized_keys

Copying public keys ¶

Copying your public key is an excellent opportunity to exercise your nascent scp skills:

$ scp ~/.ssh/id_dsa.pub ${host}:/tmp/${user}_${srchost}.pub

Good UNIX admins will create an inline loop to disseminate their keys to all systems. Once the scp is done, ssh to the remote system:

$ ssh myhost3

When first starting out, you will probably need to create a ~/.ssh directory on most systems for your target account. You can create the pub_keys subdirectory all in one command:

$ mkdir -p -m 700 ~/.ssh/pub_keys

Copy the ${host}:/tmp/${host}_${user}.pub file to your account’s ~/.ssh/pub_keys directory:

$ cp /tmp/${host}_${user}.pub ~/.ssh/pub_keys

Then, copy your public key to the administrative account public key directory:

$ sudo su - oracle
$ cp /tmp/${host}_${user}.pub ~/.ssh/pub_keys

Updating authorized_keys ¶

Updating the authorized_keys is the exact same process as described in Configuring the local account. Simply append the public key to ~/.ssh/authorized_keys:

cat ~/.ssh/pub_keys/${owner}_${host}.pub >> ~/.ssh/authorized_keys

Test access ¶

Once you’ve copied your public keys and updated the authorization file, test your access by ssh’ing to the target system from your launch system. The system should ask for your passphrase:

$ ssh myhost3
Enter passphrase for key '/home/oleary/.ssh/id_dsa': [passphrase]

At this point, you should be able to ssh into all systems as your normal user and as the application admininistrator if you so configured it. The system should be asking for your passphrase at each ssh. If you’re having issues, check the permissions on ~/.ssh (700) and ~/.ssh/authorized_keys (644 at widest, preferably 600). If both of those look right, examine syslog. Openssh is usually very good about telling exactly what’s wrong in the syslog.

The advantage to this configuration is that you can ssh into your account using your passphrase. Another benefit is that X11 is tunneled through the secure/encrypted connection. The drawback, though, is you have to supply your passphrase at every call to ssh. Depending on what you selected as your passphrase, this could be a problem. Circumventing that last issue is the subject of the next section.

Configuring ssh agents ¶

ssh-agent overview ¶

The ssh protocol provides a method of caching your private key in memory. Doing this means that your private key authentication can be done automatically vs having to supply your passphrase every time you ssh to another system. This functionality is done via the ssh-agent and ssh-add commands.

ssh-agent creates a daemon process and initializes/exports a set of environment variables that are used to create the authentication channel. ssh-add adds your identity to the agent daemon which provides your private key authentication upon request. When both of these are done correctly, the ssh authentication is tunneled back to the originating host where the ssh-agent answers. This means that you can effectively ssh from system to system to system without resupplying your passphrase.

ssh-agent initialization ¶

Officially, there are two ways to kick off the ssh-agent. There is a third way that provides even more flexibility and ease of use; however, it’s a little more complicated so we’ll start with the first two methods.

Eval method ¶

Since ssh-agent initializes and exports environment variables, you must get those environment variables into the current shell. One method of doing that is to eval the daemon process as follows:

$ eval `ssh-agent`

Please note the backticks surrounding ssh-agent. As you know from the UNIX 101 course, eval executes the program in the current shell vs starting a subshell for it.

Subshell method ¶

The other method of starting the ssh-agent is to run it in the foreground with a shell name as a command line argument. This method starts another shell, exporting the requisite environment variables. It’s syntax is as follows:

$ ssh-agent $SHELL

You could actually use this to change your shell if you wanted - simply replace $SHELL with the absolute path to the shell of your choice.

Adding identities to the agent ¶

Now that you have the ssh-agent running, you have to add your identity to the agent in order to enable the authentication. To verify that there are no keys currently installed, in the same window as you executed the ssh-agent command, execute ssh-add with a -l (lower case L) argument:

$ ssh-add -l
Listing identities.
The authorization agent has no keys.

To add your identity, execute without the -l argument:

$ ssh-add

The program will ask for your passphrase. Assuming you enter it correctly, listing the registered keys in your agent shows different results:

$ ssh-add -l
Listing identities.
The authorization agent has one key:
id_dsa_1024_a: 1024-bit dsa, oleary@myhost1, Thu Oct 10 2002 09:20:24 -0600

Congratulations! At this point, assuming you have created your keys, disseminated them, and updated the appropriate authorization files, you can now ssh/scp from system to system from this window without resupplying your passphrase at each step.

Third way of initiating ssh-agent ¶

Both methods of starting ssh-agent that we discussed bring the requisite environment variables into the current session. There is no way, using those methods, to share a ssh-agent daemon between sessions. This effectively means that every time you open a new window, you have to start a new ssh-agent daemon. It would be much nicer to the system process table if we could figure out a way to share one ssh-agent daemon across login sessions.

The key part of the trick is to realize that ssh-agent provides access to the authentication channel and all associated keys through the use of environment variables that it spits out to stdout. For instance:

$ ssh-agent
SSH2_AUTH_SOCK=/tmp/ssh-oleary/ssh2-1280-agent; export SSH2_AUTH_SOCK;
SSH2_AGENT_PID=1281; export SSH2_AGENT_PID;
echo Agent pid 1281;

This kicks off a ssh-agent daemon to which we don’t have access because the environment variables aren’t in the current session. This agent will continue running when we completely log out of the system. The only way for this daemon to die at this point is to manually kill it or reboot the system. If we execute those environment commands that ssh-agent displays via cut/paste we suddenly have access to the agent daemon. We can then add the identity key via ssh-add as normal.

$ SSH2_AUTH_SOCK=/tmp/ssh-oleary/ssh2-1280-agent; export SSH2_AUTH_SOCK;
$ SSH2_AGENT_PID=1281; export SSH2_AGENT_PID;
$ ssh-add

Now, if we open up another window and initialize/export those same environment variables, we gain access to the ssh-agent daemon running as pid 1281 and the cached private key.

Additionally, the ssh-agent daemon will now survive if we completely log out of the system. This means that we can initiate a ssh-agent once per system boot instead of once per login window.

So, how do we go about either starting a ssh-agent if it’s not already running or setting and exporting appropriate environment variables if it is? I have the following function in my myhost1:~oleary/.kshrc file:

ssh_agent()
{  Restart_agent=0
   Agent=~/.ssh/agent_info
   if [ ! -f ${Agent} ] ## Agent file not present
   then
      Restart_agent=1
   else
      Pid=$(grep -i ssh_agent_pid ${Agent} | awk -F\; '{print $1}' | \
         awk -F= '{print $2}')
      ps -p ${Pid} > /dev/null 2>&1
      if [ $? -ne 0 ]
      then
         Restart_agent=1
      else
         . ${Agent}
      fi
   fi

   if [ ${Restart_agent} -ne 0 ]
   then
      /usr/bin/ssh-agent | head -2 > ${Agent}
      . ${Agent}
      echo ""; echo "You need to add the ssh keys manually!"
   fi
}

[[ snip ]]

ssh_agent
# eof

This function checks for the existence of a ~/.ssh/agent_info file. If it’s not found, the function sets a flag variable. If the agent_info file is found, the function obtains the pid of the ssh-agent and checks to see if a process with that ID is running. If not, the function sets the flag variable. If it is, the function sources the agent_info file. The function then checks the status of the flag variable. If it’s been set somewhere in the process, the function kicks off another ssh-agent, records the environment variables to agent_info, then sources that file and alerts the user that he has to add his keys to the new agent.

With this defined in your .kshrc file, every time you log into the system, you will have access to your ssh-agent and it’s cached private key(s). No more typing your passphrase every time you ssh to a new system. So, with this in place, there is no need to use null passphrased keys for your default interactive accounts.

There is a place for null passphrased keys, though; that is the subject of the next section

Noninteractive ssh ¶

The previous sections went through configuring ssh eventually getting to the ssh-agents which support ssh/scp sessions w/o manually resupplying a passphrase. This works great for individual admins that have interactive sessions. What about root level cron jobs, Autosys, or SAP scripts that require non-interactive secure remote job execute and secure copy functionality?

Before getting into that subject, a little background:

There are three factors in authentication:

Something you know
Something you have
Something you are

The more factors you use in authentication, the more secure your session is.

Password based login methods are single factor authentication because it’s something you know (the password). The secure ID tokens used to log in to the your company’s network remotely use 2-factor authentication as it’s something you have (the token) and something you know (the pin). Public key authentication is also considered 2-factor authentication because you have the private key and know the passphrase. The third factor is generally biometrics - retinal scans, finger prints, etc.

Non-interactive ssh access is usually done through scripts or batch files via cron which has an incredibly small starting environment. We could create script logic to read in the ~/.ssh/agent_info file; however, that adds a layer of complexity that, for the most part, is unacceptable. Is the pid listed in the agent_info a ssh-agent pid? Is there a valid key in the agent? How to get a valid key if it’s not?

The commonly accepted answer to these issues is the use of null-passphrased secure shell keys. If the passphrase associated with a key pair is empty, then ssh won’t ask for it. Using these keys, though, effectively removes the “something you know” factor from the 2-factor authentication. This type of key should only be use with the following caveats:

Should only be used for non-interactive scripts
Should be locked down to the commands it needs only.
Should never be used as the default key.

You generate null-passphrased keys either by hitting return when prompted for the passphrase or by specifying it on the command line:

ssh-keygen -t dsa -f -/ego -P ""

To use the null-key, specify an alternate identity on the command line:

$ ssh .i ~/.ssh/ego ${system} ${command}
$ scp .i ~/.ssh/ego ${source} ${destination}

Miscellaneous ssh items ¶

A few additional points that don’t necessarily fit anywhere else:

~/.ssh/config ¶

You can avoid some typing by using the config file in your .ssh directory. Using this file, you can avoid constantly typing -l ${user} if you always login as a different user or even shorten long host names for ease of use. There are a fair number of configurations that can be made using this file. I generally limit it to the two mentioned above.

The syntax is pretty simple:

Host r3db17
   hostname usr3db17.ndc.myco.com
   user root
Host odie
   hostname odie.mysub.myco.com
   user root

Using this configuration, I can ssh into usr3db17.ndc.myco.com as root simply by typing:

$ ssh r3db17
# hostname
usr3db17

The second example shows how you can alias hostnames. Since the mysub.myco.com domain is not normally in the domain search path, you would have to type the entire fully qualified domain name. That’s a lot of typing, particularly if you access it regularly. The config file allows you to shorten that hostname:

$ ssh odie ls -ld /tmp
drwxrwxrwt 9 root root 8192 Jan 21 11:05 /tmp

A potential use for this, for SAP basis, would be to have aliases based on the account accessed. For instance:

host ecdadm
   hostname myhost21.ndc.myco.com
   user ecdadm
host oraecd
   hostname myhost21.ndc.myco.com
   user oraecd

That configuration file would allow the SAP Basis team to execute ssh ecdadm or ssh oraecd to access the appropriate accounts on the correct system.

ssh & clusters ¶

When you ssh to a specific host, the remote host’s key is put into ~/.ssh/known_hosts (by default). When you next connect to that host, ssh checks the host key to verify that you’re talking to the same host you were before. If the host key has changed, ssh screams about a potential man in the middle attack.

That’s exactly what you want to happen in normal circumstances. However, what happens if you’re connecting to a clustered IP address that can switch between hosts. As soon as the package fails over to another node, the host key will change and ssh will scream.

Some have suggested ensuring the nodes of a cluster share the same host key. However, that weakens the security of the cluster. What happens if you want to ensure you’re talking to a specific host? ssh won’t be able to verify that for you. While using the same host key for cluster nodes would work, there is a better way

The trick is to use hostkeyaliases in the ~/.ssh/config file and the ~/.ssh/known_hosts file.

First, configure ~/.ssh/config. For example:

host mjp
   hostname sapmjpdb.myco.com
   user oracle
   HostKeyAlias MJP

The known_hosts entry format is:

${host},${ip},${aliases},${rest-o-key}

So, you can have:

node1,192.168.12.10,MJP ssh-rsa AAA[[snip]]
node2,192.168.12.12,MJP ssh-rsa AAA[[snip]]

Once all that’s in place, you can:

ssh mjp ls -ld /tmp

and have it work regardless of the host on which sapmjpdb is currently running