SSH-from-the-ground-up

If you work professionally in the IT industry, chances are you’ve been using OpenSSH for a long time now for your day to day work.

OpenSSH however provides so much more than “just” remote shell on *nix system (and apparently on Windows too now!) and in this article
we’re going to explore some of the non immediate uses of ssh and introduce a few accessory tools that make using ssh even better.

Conventions for the examples

We need to set some terminology to avoid confusion down the road, in particular we need:

  • A client host, it will be identified simply as local, in the examples local$ will identify the shell prompt, you won’t have to copy that part of the example to try it out.
  • A few remote hosts, which will be identified as remote.domain.com. In case we’re gonna need multiple remote hosts a numeric index will be added. The prompt convention applies in this case as well.
  • Users will simply be local_user and remote_user. In case we need more users in some example, the numeric index will be added, similar to what we’ll use for the remote hosts.

The output for the examples are mostly taken from an OS X installation, but barring minor differences in default paths (for example OSX puts the home directories under /Users instead of the more common /home in linux distributions) everything else stays the same.

The basic stuff

Connecting to a remote host

Lets start small and build up on every new concept, we’re going to assume that SSH is installed on both the client and the server, for instruction on how to do so please refer to your OS documentation.
Expert users might want to skip the first couple of sections and jump right through the meaty bits.

To connect to a remote host just fire up your terminal emulator and simply connect:

`local$ ssh -l remote_user the_remote.server.com`

or more simply

local$ ssh remote_user@remote.domain.com

ssh will perform its key exchange and set up the encrypted connection, ask for a password and if everything checks out, it will let you in and will present the remote server prompt. At that stage it will be like being in front of the terminal emulator of the remote server so you can start to work away.
If your local user and remote users have the same name you can skip the username in your ssh invocation.

Configuration, part one

Realistically, you do not want to have to type the fully qualified hostname every time you connect on a machine, so it is possible to define short names through the ssh client configuration.

On *nix system, the client configuration generally resides in two places:

  • The global ssh configuration in /etc/ssh/ssh_config
  • The user configuration in $HOME/.ssh/config

The format and the options are the same for both files as detailed in the ssh_config manual page.

Let’s take a look at a simple configuration:

 Host remote 
 HostName remote.domain.com 
 User remote_user

With these two lines in place you can just ssh remote. The ssh client will read the configuration and connect to remote_user@remote.domain.com. In case the DNS record for the remote host is not set up, an IP address can be specified in the Hostname directive.

Public key authentication

By default ssh is going to accept password based authentication, but a better authentication with asymmetric keys is available and provides better security than passwords.

The increased security is given not only by a greater entropy in the generated keys when compared by a password, but most importantly by the fact that the private part of the key is never sent on the network.

Let’s generate a new ssh key using the ssh-keygen tool

local$ ssh-keygen
Enter file in which to save the key (/Users/local_user/.ssh/id_rsa): /Users/local_user/.ssh/blog_key #replace this with your own path or leave the default
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /Users/local/.ssh/blog_key.
Your public key has been saved in /Users/local/.ssh/blog_key.pub.
The key fingerprint is:
SHA256:mA/0ip4RUojNRtJK0aZhyJr7HaS1aOExLZzcqApeqC0 local_user@local
The key's randomart image is:
+---[RSA 2048]----+
|+o+              |
|.@.+             |
|==O=. .          |
|+oX.=. +         |
| +oX..+ S        |
|+.=ooo +         |
|=+..o.. .        |
|E.o..o           |
| .  o            |
+----[SHA256]-----+

As the output mentions two files where created inside the ~/.ssh directory, a private part (blog_key) and a public part (blog_key.pub). The public part is the one that needs to be distributed to the remote hosts.
By default ssh-keygen generates 2048 bit RSA keys, which are generally safe enough for normal use, however you might want to have a stronger key, and it can be obtained by adding the -b 4096 option which will create a 4096 bit key.
It is also possible to have different key types, such as ecdsa or ed25519, simply add the -t switch followed by the type of key you’re interested in.

The public keys need to be added to the ~/.ssh/authorized_keys on the remote hosts, lets copy it manually on our remote:

local$ cat  /Users/local/.ssh/blog_key
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRQ6CjxzskkXQtC9H2cV9jGt8T8kmZHeLXNrt5drO0PuKeA2VvmG96m5EFCgKTR5Rmug2poFy7QvRlyuwauEjPm0cImX1VpVKNO4GTo7PmEmt1MwvkcSzMb2U5AgXIeth7lxtZ0H0jOaW3371xfyaHn0L/LXbZyFOiHkz/TreNg0Mj1FatX8nQYSNaQwD1byTAu2Z8WENJ0JY26zmbMr1hqKYXTJ5GZ8NkqK5VmW+zQ/wnOjMaKz9IchULHfedaakDUVWFY1hailOzhHU+H32gFZLFneHG1mlKQQ3P3TCWmecfG5mARLPrlamR2UvLSZin4LOi6XwSfJ4eWcUQlcg/ local_user@local

local$ ssh remote
Password:
remote$ mkdir -p ~/.ssh; chmod 700 ~/.ssh
remote$ echo "$content_of_the_public_key_file " >> ~/.ssh/authorized_keys

Done! the next ssh session to remote will not ask you the password for the remote user but instead, if set, the passphrase of the key (it won’t prompt for anything if the passphrase has not been set).

To simplify things, most linux distributions ship with an utility script called ssh-copy-id which performs exactly the same thing in the example above, but in an automathed way, simply invoke ssh-copy-id remote. On OSX this utility is missing by default but can be installed. If you have multiple keys on your system you can use the -i option, passing in the path to the key you want to install on the remote host.

“Passwordless” ssh

We now have a configuration, a public key that is installed on the remote host but we still have to type the key’s passphrase every time we open a new connection. If you find yourself opening tons of connections to many hosts it gets old quite fast, and one might be tempted to remove the passphrase from the key which is of course not a great idea.

Luckily the OpenSSH developers come to the rescue again with ssh-agent.
ssh-agent is a client program that lives on your local machine and acts as an authentication agent for every ssh client invocation, allowing the agent to securely store in memory an unecrypted version of your key once you add it to the agent.

To start the agent simply run:

local$ eval $(ssh-agent)

or if you prefer, run ssh-agent and then export the SSH_AGENT_PID and SSH_AUTH_SOCK variables that ssh-agent printed out when it started.
Those two environment variables are checked by the ssh client to know where the agent lives and which socket to use.
By default the agent socket is created under a temporary directory, but it’s possible to ask the agent to set it into a specific path using the -a option, which is useful to automagically reuse an agent between user sessions on the local machine. More on this later.

Once the ssh agent is running, we can add our key to it by running:

ssh-add ~/.ssh/blog_key
Enter passphrase for /Users/local_user/.ssh/blog_key:
Identity added: /Users/local_user/.ssh/blog_key (/Users/local_user/.ssh/blog_key)

and to verify that our key is present and enabled:

local$ ssh-add -l
2048 SHA256:mA/0ip4RUojNRtJK0aZhyJr7HaS1aOExLZzcqApeqC0 /Users/fpedrini/.ssh/blog_key (RSA)

At this point, a connection to our remote host will happen automatically without asking for any password, the authentication has been handled by the agent.

We’ll get back on this argument later in the article.

Tunnels

One of the most appreciated features of SSH is the ability to forward traffic in a variety of ways, local and remote tunnels and dynamic proxying.

Local forwarding

Lets assume that on remote there is a service listening on a specific port but only on the local interface but it would be extremely useful to be able to connect to it from the local machine. An example is trying to connect to a database instance with a local client, for this example we’re going to use PostgreSQL, but really any other service that listens on a TCP port can be exposed to the local machine using an ssh tunnel.

For example the following will not work:

local$ nc -vz remote.domain.com 5432
nc: connect to remote.domain.com port 5432 (tcp) failed: Operation timed out

ssh can come to the rescue with the -L option:

local$ ssh -Llocalhost:5432:localhost:5432 remote

At this stage, every TCP connection to port 5432 on the local machine will be forwarded to port 5432 on the remote host.
Unfortunately the user experience for tunnels in ssh is not quite straight forward, lets dig into the details.

-L is simply the the switch for enabling a local forwarding tunnel, the following is the definition of the tunnels which can be read as:

local_bind_address:local_port:remote_bind_address:remote_port

The the first two values refer to the local end of the tunnel, where ssh should setup a listening socket to accept incoming connections, the last two are the remote end, so where ssh needs to forward the traffic coming from the tunnel.

In particular the local_bind_address can be omitted and it’s set by default to the loopback interface of the client machine, but it is possible to specify a public interface.
Similarly we can do the same for the remote_bind_address, which can even be a third host that is reachable only from the remote host.

Following the current example on our local host we’ll be able to see a process listening on port 5432:

local$ lsof -n -iTCP:5432 | grep LISTEN
ssh     14443 local_user    9u  IPv6 0x39970c5c548f41ff      0t0  TCP [::1]:5432 (LISTEN)
ssh     14443 local_user   10u  IPv4 0x39970c5c5631601f      0t0  TCP 127.0.0.1:5432 (LISTEN)

which means that the tunnel is established. Barring any firewall rules on remote, we should be able to perform at least a tcp connection to port 5432 on localhost, and have that connection forwarded to the remote server on the same port:

local$ nc -vz localhost 5432
Connection to localhost port 5432 [tcp/postgresql] succeeded!

It works! But what about a real PostgreSQL client connection?

local$ psql -H localhost -U dbuser
Password for user dbuser:
psql (9.4.5, server 9.4.5)
SSL connection (cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256)
Type "help" for help.
dbuser=> 

Of course there is no need to use the same port on both the local and the remote machine, but it often makes sense, especially when using services with clients that by default connect to a predefined port (such as in postgre’s case), otherwise you’ll have to instruct your client to connect to a different local port.

As all options in ssh command line, this is exposed also through the configuration file, the directive is called LocalForward and it accepts the same parameters as the command line, so the following configuration will always open a local tunnel for port 5432 every time a new connection to remote is started:

 Host remote
     HostName remote.domain.com
     User remote_user
     LocalForward localhost:5432:localhost:5432

Remote forwarding

Sometimes it’s useful being able to do the opposite of a local forward, that is being able to expose a port on a remote host that leads right back to a port on the local machine.
This is especially useful when the local machine is behind a firewall and it’s not directly exposed on the Internet. Of course “evading” the firewall through ssh to expose non whitelisted ports might be in violation of company policies, so please be careful when using this in a professional setting :).

The option to enable remote forwarding works pretty much in the same way as the local forward, but the arguments are swapped, the first bind_address:port pair refers to the remote machine and the second one to the local machine.

For this example we’re going to try something that might seem weird at first, but has it proven to be a lifesaver in more than a few occasions.
Specifically I used this trick time and time again to access my home network, which is sadly behind a NAT outside my control. In this case remote is a server which is remotely accessible through SSH.

local$ ssh remote -R localhost:1234:localhost:22

Great! now leave the session open and from a shell on the remote box (either through ssh or directly on the remote‘s console) try an ssh connection to localhost on port 1234:

remote$ ssh localhost -p 1234
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is 50:22:3d:bc:a7:02:45:e1:a0:1e:df:38:0f:85:6f:f9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
local$

As you can see, remote was able to access local which is behind a NAT.
Of course all this works as long as the initial connection set up by local stays up, but using a tool such as autossh can help in ensuring a reliable connection.
It is also possible to tell the remote that other hosts can access the forwarded port, but for that you’ll need to modify the ssh server on remote to allow it. The option is called GatewayPorts and by default is turned off, for a number of valid security reasons. It might totally be fine if the remote is only accessible through a trusted network, but I would not recommend this solution, the security risks are just too high and there are better ways to achieve the same result (such as a properly set up VPN server).

Similarly to what happens for local forwards, we can configure ssh to open a remote forward on every connection using the RemoteForward configuration option.

Again, please be careful when using this in a professional setting, it might be considered unauthorized access and cause you lots of troubles.

Dynamic proxying

Building on top of local and remote forwards, SSH is also able to act as a dynamic proxy, which can be considered sort of a poor’s man VPN. With an ssh dynamic proxy set up (and your browser configured to use it), all your connection to websites will be routed through the remote box.
This is as simple as running:

local$ ssh remote -D 8080

The ssh client will start listening locally on port 8080 (or whatever port specified after -D) and expose a SOCKS Proxy, the next step would be to configure the browser to use it.
There are too many browsers and extensions to manage proxy setup to provide a meaningful overview, refer to your browser documentation for it, just know that the type of proxy is SOCK5 and the address will be localhost:$PORT.

Connection multiplexing

SSH is able to set up connection multiplexing, creating a local socket for each host and reusing it in case a new session is open to the same box. This provides some speedups in the setup of the connection, or more precisely does not set up a connection at all, it just communicates to the remote ssh server to open a new user session with the same parameters as the already established one.

This requires a directory on the filesystem where the socket will be created, I personally use ~/.ssh/sock but any directory owned by your user and with restricted permissions (0700) will do.

local$ mkdir -p ~/.ssh/sock
local$ chown 700 ~/.ssh/sock

Then edit the configuration and add the following:

 Host * 
 ControlMaster auto 
 ControlPath ~/.ssh/sock/%r@%h:%p

The first line is just the definition of the ‘ssh host’, but instead of using an identifier it uses a pattern, in this case this configuration will be applied to all hosts.
It is possible to restrict the pattern (such as *.domain.com) or just specify these options for a single host.

The ControlMaster line, instructs ssh to try and reuse a master connection if it exists and create one in case it does not.
ControlPath instead specifies where the socket file must be created. The weird part at the end of the paths are placeholders, specifically in this example:

  • %r is the remote user
  • %h is the remote host
  • %p is the remote port

which means that invoking ssh remote will create ~/.ssh/sock/remote_user@remote:22.

There are many options for substitution documented in the ssh_config manual page. It is recommended to at least specify the three placeholders mentioned here, which should suffice to uniquely identify a connection to a host.

Bastion Hosts and ProxyCommands

It is normal practice to expose a remote infrastructure through bastion hosts, that is a server which exposes ssh on the internet and allows access to the servers behind it, which are not exposed. In the example I’m going to refer to this host simply as bastion.
Lets say that we have a bunch of servers hosted somewhere and exposed through bastion.domain.com, and we want to access a remote behind it. A way to do it is to simply connect to the bastion and from there start a connection to the destination host:

local$ ssh bastion.domain.com 
bastion$ ssh remote.domain.com 
remote$

But again, it is something that gets old really fast, luckily ssh provides a ProxyCommand option which helps in automating the process. ProxyCommand is just a generic way to override the default connection mechanism in ssh and pipe it through an external command (which can be ssh itself, as in the following example).
To enable the use of a bastion host through ProxyCommand add something similar to your ssh configuration:

 Host *.domain.com 
 ProxyCommand ssh bastion.domain.com -W %h:%p

%h and %p have the same meaning as before, remote host and remote port (they’re in fact default placeholders that are recognized by every option that deals with hosts) and the -W option tells the ssh client that it should forward standard input and standard output to the remote host.
Assuming the bastion and the remote hosts are already set up for key authentication as described earlier, ssh remote.domain.com will automatically connect to the bastion host, which will open a connection to remote and forward standard input and output to it.

There are also lots of neat tricks you can do with ProxyCommands since ssh just executes whatever command you specify in the configuration as a normal process and it even goes through variable expansions.

For example, having to specify domain.com for every ssh connection gets in the way after a while, so I have set up my ssh configuration to access my own personal servers behind a bastion host in this way:

 Host *.home 
 ProxyCommand ssh bastion.mydomain.com -W $(basename %h 
 .home):%p

And I invoke ssh using ssh host.home. basename will strip the .home suffix and forward stdin/stdout from the bastion host to host. The bastion is set up to use the local DNS search path, so that I can use non fully qualified hostnames inside my home network.

Another example is to route SSH connections over TOR:

local$ ssh -o "VerifyHostKeyDNS=no" -o ProxyCommand="nc -X 5 -x localhost:9150 %h %p" remote.domain.com

In this example we’re using netcat to proxy the ssh connection to the Tor server listening on port 9150 and we’re disabling the Dns host key verification to avoid any potential leak of information outside the Tor network.

SSH escape characters

Often times it happens to me to open a quick ssh session to verify something or run a couple of quick commands… Fast forward 2 hours later I’m still connected to the box, typing furiously on the keyboard with obscure environment variables set,a few background processes running and of course none of them through tmux or screen (it was a five minute job after all), and it’s now 3AM in the night and I notice that I really need a local tunnel to complete what I’m doing… But I don’t want to lose any of the processes I have running nor do I want to mess around with things like disown and retty

To open a local tunnel I’d have to disconnect, lose everything in the background, reconnect with the forwarding enabled and re-setup my environment, which is painful (especially at 3AM when these sort of things happen to me)… Luckily the OpenSSH developers thought of such possibility and they have introduced a bunch of escape sequences in the ssh client.

The ssh escape sequences are prefixed by a tilde (~) character, and the ~? displays the help for all the escape characters:

local$ ssh remote
remote$ ## press in sequence <ENTER>~?
Supported escape sequences:
 ~.   - terminate connection (and any multiplexed sessions)
 ~B   - send a BREAK to the remote system
 ~C   - open a command line
 ~R   - request rekey
 ~V/v - decrease/increase verbosity (LogLevel)
 ~^Z  - suspend ssh
 ~#   - list forwarded connections
 ~&   - background ssh (when waiting for connections to terminate)
 ~?   - this message
 ~~   - send the escape character by typing it twice
(Note that escapes are only recognized immediately after newline.)

The escape sequences I use the most are ~C and ~# and sometimes when I’m dealing with unresponsive clients ~. which terminates the connection comes in handy as well.
Worth noticing that escape sequences are recognized only after a new line.

The beauty about ~C is that it allows to open and remove port forwards without having to break connection. In the command line the format is exactly the same as the client options, lets see a practical example:

Lets start a connection with no forwards enabled and then enable a local forward to port 5432 (similar to the earlier example):

local$ ssh remote
remote$ ## type <ENTER>~C
ssh> ?
Commands:
      -L[bind_address:]port:host:hostport    Request local forward
      -R[bind_address:]port:host:hostport    Request remote forward
      -D[bind_address:]port                  Request dynamic forward
      -KL[bind_address:]port                 Cancel local forward
      -KR[bind_address:]port                 Cancel remote forward
      -KD[bind_address:]port                 Cancel dynamic forward
      
remote$ ## type <ENTER>~C again
ssh> -L 5432:localhost:5432
Forwarding port.
remote$

As you can see from the help message of the built in command line, it is possible to enable all kinds of forwards, using the same syntax that you would use on a normal ssh invocation.

Some tips and tricks

Now that we went through various features, lets see how to put them to use with a few tips and tricks.

Copying files

SSH comes with a tool for remote file transfer called scp which uses the ssh transport protocol behind the scenes and it’s able to use the same options specified in the ssh configuration.
scp tends to mimic cp for remote files, the basic example is the following:
scp local_file remote:/remote/path
The file will be transferred and to the remote host in the path specified after the : character. It is also possible to preserve permissions and access and modification times with the -p switch.

But what if you want to copy files between two remote hosts? Just specify two remote servers one as the source and the other as destination:

scp remote1:/path/to/source/file remote2:/path/to/destination

Running one shot commands

With ssh it’s also possible to run one shot commands without leaving a shell open, for example:

local$ ssh remote uptime
 16:25:04 up 15 days,  6:28,  0 users,  load average: 0.00, 0.01, 0.05
will show the output on `uptime` on the remote host.

This is especially useful to run the same command on a small set of hosts, for example to obtain the uptime output for 10 remote hosts (remote1 to 10 in the example below) one might use something similar to the following:

for i in $(seq 1 10); do ssh remote${i} uptime;done

The output will consist of 10 uptime outputs, one per host.
This also means that you can pipe the output from ssh to other commands using the normal unix pipes.

It is also possible to run complex commands through ssh, but to avoid problems with pipes and quotes that will inevitably get you confused when the local shell tries to interpret some of them, I prefer to put the set of commands in a text file and then invoke ssh in this way:

ssh remote $(cat command_file.txt)

The file will be read by the shell (through command substitution) and passed in to ssh as an argument, which will forward the commands to the remote host.

Automatically starting a screen (or tmux) sessions or reattach to an existing one

Given that we can run “one shot commands”, we can easily extend this to connect to a host and automatically starting a screen session:

local$ ssh remote screen -xRR

or with tmux:

local$ ssh remote tmux a -d

Of course if you use named sessions, do not forget to add the session names to the command 🙂

Fixing Remote host verification failed messages.

Every time you connect to a new host, ssh records a unique cryptographically secure fingerprint presented by a server, and it checks that the key presented by the remote is the same as the recorded one. If that’s not the case ssh halts the connection to avoid falling for a Man In The Middle attack.
However it happens that a remote host is rebuilt, or for some reason the server key is regenerated, especially when working with cloud infrastructures, where capacity is elastic and hosts are rebuilt frequently.

If you’re sure that the host key was changed and you still want to connect, you can remove the old key from the known_hosts file by using the -r switch of ssh-keygen:

local$ ssh-keygen -r remote.domain.com
# Host remote.domain.com found: line 103
/Users/local_user/.ssh/known_hosts updated.
Original contents retained as /Users/local_user/.ssh/known_hosts.old

Agent forwarding and ssh-ident

Earlier in the article we saw how ssh agent holds your keys securely in memory on your local machine, but it is also possible to export the agent to a remote host and be able to use the keys from your agent to log on other remote hosts:

local$ ssh remote -A
remote$ ssh-add -l 
2048 9c:e3:59:71:13:75:fb:24:e8:39:dd:ba:a0:3e:33:67 /Users/fpedrini/.ssh/blog_key (RSA)

-A is equivalent to specifying ForwardAgent yes in the ssh config.
If we hadn’t specified -A in the ssh invocation, the result would have been an error mentioning the fact that there are no ssh-agents instances accessible on the remote host.

SSH agent forwarding however, comes with a major security drawback. Agent forwarding works by creating a socket on the remote host that is used to communicate (through the ssh channel) with the agent on the local machine. This socket is created under /tmp and is accessible only by the remote user and root.
This means that if there is any attacker (or playful colleague?) that has root access to the remote machine he will have access to all your ssh identities for the duration of your ssh connection.
Needless to say if you have multiple ssh keys for different kind of hosts, the attacker will have access to all of them which is not the best of things…
A better solution would be to be able to have multiple ssh keys (one per environment/system/customer whatever makes most sense) and being able to forward only the relevant identity, this way the risk is not completely eliminated but the effect of a breach of the forwarded agent is drastically reduced.

ssh-ident is designed for exactly this purpose, it’s an ssh-agent manager, that spins up agents and loads identities on demand the first time they’re needed (and by default unloads them after a set timeout).

ssh-ident is a wrapper around ssh, to use it just set an alias that overrides it:

alias ssh='/path/to/ssh-ident

and then call ssh as normal, ssh-ident will spin up an ssh-agent and by default try to load the default key:

local$ ssh remote
Loading keys:
    /Users/local_user/.ssh/id_rsa
Enter passphrase for /Users/local_user/.ssh/id_rsa:
Identity added: /Users/local_user/.ssh/id_rsa (/Users/local_user/.ssh/id_rsa)
Lifetime set to 7200 seconds
remote$

At this point setting up new identities is a matter of creating as many keys as you need and saving them in ~/.ssh/identities/, here is a quick example with two keys, one for “work” and one for “home”:

mkdir -p ~/.ssh/identities;chmod 700 ~/.ssh/identities

ssh-keygen -f ~/.ssh/identities/work
ssh-keygen -f ~/.ssh/identities/home

and drop the following in the ssh-ident configuration file (~/.ssh-ident):

MATCH_ARGV = [
    (r"home", "home"),
    (r"domain.com", "work"),
]

This configuration tells ssh-ident which identity to use for hosts that mache ‘home’ or ‘work’. There are multiple ways to specify which identity to use (and also lots of ways toadd ssh-options on the fly) as mentioned in ssh-ident’s README, and every user should find its own way to map identities to hosts.

Lets see what happens if we ssh to a ‘home’ host without ssh-ident, for example purposes I’ve already setup a local agent and added both keys generated previously:

local$ /usr/bin/ssh remote.home
remote$ ssh-add -l
2048 de:df:85:53:9d:87:cb:c7:c1:17:f7:10:7a:41:e0:54 /Users/local_user/.ssh/identities/work (RSA)
2048 9c:e3:59:71:13:75:fb:24:e8:39:dd:ba:a0:3e:33:67 /Users/local_user/.ssh/identities/home (RSA)

As you can see, both ssh keys are forwarded as expected. Which means that I’m technically leaking a work related key on my home network, and conversely when I connect to a host at work I’m leaking my home key.

Now let’s try with ssh-ident:

local$ ssh remote.home
Loading keys:
    /Users/local_user/.ssh/identities/home
Enter passphrase for /Users/local_user/.ssh/identities/home:
Identity added: /Users/local_user/.ssh/idenities/home (/Users/local_user/.ssh/identities/home)
Lifetime set to 7200 seconds
remote$ ssh-add -l
2048 9c:e3:59:71:13:75:fb:24:e8:39:dd:ba:a0:3e:33:67 /Users/local_user/.ssh/identities/home (RSA)
remote$ logout
local$ ssh remote.work
Loading keys:
    /Users/local_user/.ssh/identities/home
Enter passphrase for /Users/local_user/.ssh/identities/work:
Identity added: /Users/local_user/.ssh/idenities/work (/Users/local_user/.ssh/identities/work)
Lifetime set to 7200 seconds
remote$ ssh-add -l
2048 de:df:85:53:9d:87:cb:c7:c1:17:f7:10:7a:41:e0:54 /Users/local_user/.ssh/identities/work (RSA)

Quite neat!

Of course the forwarded key is still exposed to all users that have root access on the remote machine, but at that point an eventual attacker will be able to compromise only one key instead of all the identities in your ssh agent.

Conclusions

This article got quite long, but it just manages to scratch the surface of the almost countless ssh options and weird stuff that might be done with it, but hopefully gives enough information to increase productivity and security on the day to day life.

For any other option, I strongly recommend reading the wonderful manpages that comes with ssh, there are plenty more gems in there to be discovered and the number of useful features grows with every OpenSSH release.