Xensoft

Become an SSH Ninja: SSH Tips and Tricks

John Westlund -

OpenSSH has some pretty cool tricks up its sleeve beyond it's "Clark Kent"-esque ability to provide a secure command line interface on a remote system. Here are some of its more "Superman"-type abilities:

Public-Key Authentication

This really is a best practice and can add another layer of security to your systems. Most systems with sshd running will allow you to log in using a public/private keypair. To generate one:

$ ssh-keygen -t rsa

Here you'll have the option to add a pass phrase (extra security) to your key pair, or go password-less (convenient). Omitting a password means you can log into any machine you've copied your public key to without typing in a password, pretty convenient as long as your private key stays secure. What you may want to do is set your account's password to something extremely strong, such as a pass phrase, and then use a shorter but still strong password for your key pair.

After ssh-keygen is finished you'll have a public (~/.ssh/id_rsa.pub) and private (~/.ssh/id_rsa) set of keys that you can use to authenticate to a remote system. Add the contents of your public key to your authorized_keys (~/.ssh/authorized_keys) file on all the systems you want to easy access to and you're set!

Since this is a pretty routine thing to want to do you can easily move your public key to a new system using ssh-copy-id:

$ ssh-copy-id remoteuser@remotehost

ssh-copy-id ships with most openssh-client packages, or on a Mac can be installed using brew. You can install Homebrew per the instructions at http://brew.sh/. You can find alternate instructions here.

Learn to Love Your .ssh/config

You can accomplish all of these tricks on the command line and even make them simple to use by creating an alias, but you'll be missing out on taking advantage of these options with other ssh-friendly commands (e.g. scp, rsync, etc). The syntax:

Host hostname
    Option value

The Host value refers to the alias or DNS name you'll be referring to on the command line, and doesn't need to match the actual hostname of the remote system:

Host work
    HostName timesink.initrode.com

Now I can ssh to "work" and ssh will know I'm trying to connect to timesink:

# This:
$ ssh -p 2000 -i ~/.ssh/workkey stooge@timesink.initrode.com
# Becomes:
$ ssh -p 2000 -i ~/.ssh/workkey stooge@work

But I can clean things up even more!:

Host work
    User stooge
    Port 2000
    IdentityFile ~/.ssh/workkey
    HostName timesink.initrode.com

Now suddenly connecting to timesink.initrode.com becomes:

$ ssh work

You can even have options that will apply to all hosts you connect with:

Host *
    <default options>

Options are read by ssh in the following order:

  1. command line
  2. user's configuration file (~/.ssh/config)
  3. system-wide configuration file (/etc/ssh/ssh_config)

ssh will use the first instance it sees of an option. Therefor options on the command line will always take precedence. You'll want to put your default configuration options (Host *) at the end of your ~/.ssh/config so that any specific host options are processed first.

Agent Forwarding

Now that you've got a public/private key pair it's time to copy them everywhere, right? That's one solution if you want to easily be able to log in from system to system, but you'll want to protect your private key like your password and limiting the number of systems it lives on is a good practice:

fortknox$ ssh sketchy1  # our private key lives in fortknox
sketchy1$ ssh sketchy2  # our public key is here, but we don't want to put our private key out here
Password:               # without our private key we'll have to use another authentication method

However, since our private key is available upstream on fortknox we can configure ssh to forward our credentials down the chain of connections. This is known as "Agent Forwarding." First make sure your ssh-agent is running:

$ ssh-add -L
Could not open a connection to your authentication agent.

If you saw the above then either your environment isn't directed you to your already running ssh-agent or it's not running. Ideally it will get launched automatically for you when needed, so add the following to your .bashrc:

SSH_ENV="$HOME/.ssh/environment"

function start_agent {
    printf "Initializing new SSH agent..."
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo succeeded
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add;
}

# Source SSH settings, if applicable

if [ -f "${SSH_ENV}" ]; then
    . "${SSH_ENV}" > /dev/null
    #ps ${SSH_AGENT_PID} doesn't work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi

Start up a new shell and you should see:

Initializing new SSH agent...succeeded

Now ssh-add should show:

$ ssh-add -L
The agent has no identities.

Time to add some identities:

$ ssh-add
Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa)

Now we need to tell ssh to forward our credentials, to make this automatic add the following to your ~/.ssh/config:

Host *
    ForwardAgent yes

You'll also need to make sure that the sshd on sketchy1 and the rest of the systems you want to use as intermediaries has the following in their /etc/ssh/sshd_config files:

AllowAgentForwarding yes

Now your private key is safe and sound on fortknox, but you get all the advantages of having your private key out among all the sketchy hosts.

Sharing Connections

If you spend a lot of time connecting/reconnecting to hosts with long complicated passwords or that use some kind of One Time Password (OTP) authentication you can take advantage of ssh's ability to create multiple sessions over a single network connection.

The secret? Use ControlMaster:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/master-%l-%r@%h:%p

With the above in your ~/.ssh/config or /etc/ssh/ssh_config when you connect to any host (hence the * after Host, feel free to change this to a particular host that you'd like the following configuration stanza to apply to) ssh will create a control socket using the syntax provided in ControlPath. The various variables available for naming your control socket are:

%L - first component of the local host name
%l - local host name (including any domain name)
%h - target host name
%n - original target host name specified on the command line
%p - the port
%r - remote login username
%u - username of the user running ssh

This allows ssh to uniquely identify each control socket.

Setting ControlMaster to auto makes ssh opportunistic about creating control sockets. Initially it will try to connect over an existing control socket, if none exists it will create one once you've created your initial session.

What's the upshot? After you've created your first session every subsequent attempt to connect to the same host via ssh (this includes commands that take advantage of ssh like scp!) Will use the already existing network connection, creating a new session (or copying a file, etc) without re-entering your credentials.

Bonus

If you find yourself disconnecting from a host, only to then realize you needed to do one last thing then you can take advantage of your ControlMaster configuration to keep a background connection open to a remote host even after the initial connection has closed:

Host *
    ControlMaster auto
    ControlPath ~/.ssh/master-%l-%r@%h:%p
    ControlPersist 600

Now even when you've closed your initial session to a remote host the network connection will remain open for 10 minutes (600 seconds). If the control socket remains idle for that period then ssh will clean up the connection. You can adjust the timeout or set the value to "yes" and ssh will keep the connection open indefinitely unless manually dropped or the connection is lost.

Use Tab-Completion with Hostnames

Most relatively modern shells will allow you to use tab-completion with the hosts in your ~/.ssh/known_hosts file. Now with a few quick key presses:

# This:
$ ssh whydidimakethishostnamesolongquestionmark.initrode.com
# Is as simple as:
$ ssh why<tab><tab>
whydidimakethishostnamesolongquestionmark.initrode.com whywouldyouhaveanotherlonghostname
$ ssh whyd<tab>
$ ssh whydidimakethishostnamesolongquestionmark.initrode.com

If you're using bash and not seeing this behavior you can try the following:

function autoCompleteHostname() {
  local hosts=($(awk '{print $1}' ~/.ssh/known_hosts | cut -d, -f1));
  local cur=${COMP_WORDS[COMP_CWORD]};
  COMPREPLY=($(compgen -W '${hosts[@]}' &mdash; $cur ))
}

complete -F autoCompleteHostname ssh

Tunneling

I find myself using this mostly for web servers that live behind firewalls, or are only listening locally on a remote system. If you need to access a port that it not externally available to you then you can ssh to a system which does have external access, or even the system itself and forward that port back over ssh.

$ ssh -L 8080:firewalledhostname:80 remoteuser@remotehost

Now point your local browser to http://localhost:8080 and you're golden.

Remembering our love for our ~/.ssh/config we can accomplish the same thing via:

Host remotehost
    LocalForward 8080 firewalledhostname:80
    User remoteuser

...in the Background

Often when tunneling a port your don't really need or want a shell on the remote system. ssh has you covered:

$ ssh -f -N remotehost

-f puts ssh in the background before command execution, and -N tells ssh to not run a remote command.

Now in Reverse

Sometimes because of firewall rules, etc. you can only initiate your ssh session from within a network, but your still want to be able to forward a port back into the intranet where you originated your ssh session, enter Remote Port Forwarding:

ssh -R 8080:intranethost:80 remotehost

Same basic syntax as the Local Port Forwarding we did in Tunneling except now it's remotehost that has access to the forwarded port on the side we originated our ssh session from. Openinig a connection on remotehost to port 8080 would grant access to port 80 on intranethost. Adding this to our ~/.ssh/config is as easy as:

Host remotehost
    RemoteForward 8080 intranethost:80

Lightweight SOCKS Proxy

Tunneling solves the case where there is a particular host you're trying gain access to, but if there is a range of hosts behind a firewall or router that you can't access without being internal to the network you can use ssh as a lightweight proxy (also known as Dynamic Port Forwarding):

$ ssh -D 3128 remoteuser@remotehost

Then in your browsers proxy settings you can add a SOCKS proxy on localhost port 3128:

If the DNS name of your remote systems are only known on the internal network you may have to tell your browser to use remote DNS over the proxy. In Firefox you'll have to change the setting under http://about:config :

Or use the amazing FoxyProxy.

Mount Remote Directories/Filesystems Locally

Rather than shuffling files back and forth between a remote and local system you can bring the remote directory to you with the help of sshfs. You may not have sshfs installed by default but it should be available through your package manager of choice and on Mac's its available easily through homebrew:

$ brew install sshfs

With sshfs at our disposal those remote files are easily available by running the following:

$ sshfs -o idmap=user remoteuser@remotehost:/usr/local /mnt

Everything works like you would expect from a normal mount command. The only extra tidbit is the option passed: -o idmap=user this will attempt to map your local numeric UID to the remote system's numeric UID for remoteuser.

Once you've finished working on the remote files and it's time to unmount:

$ fusermount -u /mnt

Editing Remote Files

You're going to want to have keys or ControlMaster configured for this or you'll be typing your password in regularly, but vim is capable of editing files over scp:

vim scp://remotehost/remotefile

The path of remotefile is relative to the path of your home directory on the remote system, so in the example above you would likely be editing: /home/username/remotefile

You can take this trick a bit further because what we're using above is vim's netrw capability which allows vim to navigate remote and local file systems. Just like pointing vim to a local directory, if you point vim at a remote directory you can navigate the file system and find the files you're looking for and then start editing them:

$ vim scp://remotehost/remotedir/
" ============================================================================
" Netrw Directory Listing                                        (netrw v140)
"   /home/remoteuser/remotedir/
"   Sorted by      name
"   Sort sequence: [\\/]$,\\<core\\%(\\.\\d\\+\\)\\=\\>,\\.h$,\\.c$,\\.cpp$,*,\\.o$,\\.obj$,\\.info$,\\.swp$,\\.bak$,\\~$
"   Quick Help: <F1>:help  -:go up dir  D:delete  R:rename  s:sort-by  x:exec
" ============================================================================
../
.adobe/
.cache/
.config/
...

And Now with Emacs

It's not my cup of tea, but you can also edit remote files using emacs. Just open a file using the following syntax: //remoteuser@remotehost:/remotefile

Access Remote Internal Hosts Easily

If you routinely need to log into system that are NAT-ed or only accessible via another system you could set up an ssh tunnel like what I described earlier:

$ ssh -f -L 2222:hiddenhost:22 gatewayhost -N
$ ssh -p 2222 localhost

But we can also tell ssh how to proxy our connection via another host, also known as a jump host:

Host hiddenhost
    ProxyCommand ssh -q -W %h:%p gatewayhost

The ProxyCommand tells ssh how it should connect to hiddenhost. In this case by ssh-ing to gatewayhost an then forwarding traffic via the -W flag (which forwards STDIN and STDOUT to the given host:port). I could have listed hiddenhost:22 after -W here, but instead I've used the variables %h and %p which ssh interprets as the host and port I'm trying to connect to.

Now you no longer have to stage files on gatewayhost prior to copying them to hiddenhost. You just have to remember if you're prompted for a password it could be for gatewayhost or hiddenhost.

One caveat, if you're using ControlPersist then this will fail with a message about a garbled message. I haven't found a reasonable work around yet, so pick one of the other.

SSH Escape Sequences

Using the tilde (~) after hitting [enter](or [return]) you can access various features:

Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

From here you can terminate your session (and unfortunately any shared connections). In fact most of these are unavailable during shared (multiplexed) connections.

One of the cool things you can do if you're not sharing your connection is add options you forgot when you first made a connection, such as port forwarding:

$ ~C
ssh> -L3128:localhost:3128
Forwarding port.

Boom. Port forwarded after the fact.

You can also quickly get back to the host you came from by back-grounding your session:

remotehost$ ~^Z [suspend ssh]

[1]+  Stopped                 ssh remotehost
<run essential local commands>
localhost$ fg
ssh xenmini
remotehost$

Good stuff.