Open in app

Sign in

Write

Sign in

AWS SSM HTTPS/SSH Reverse Tunnel

Glen Tomkowiak
6 min readSep 22, 2018

The Amazon AWS Systems Manager Agent (SSM Agent) is a great way to manage systems in EC2 or on premises. It can run shell commands remotely and return a response. But sometimes there is no substitute for a full SSH session.

The problem is most firewalls will block incoming or outgoing SSH connections because of the security risk. This guide shows a simple way to trigger a reverse tunnel with SSH over HTTPS back to an EC2 instance you can use to remotely control a system.

All SSH keys are generated on demand and never reused. The remote connection will use a service account that does not provide a shell. Keys are removed after the connection is made so the tunnel cannot be easily hijacked.

This guide assumes you are using Ubuntu Linux 16.04 LTS on the server and remote client.

1. Set up an EC2 Apache2 server for SSH over HTTPS

First you will need something to connect back to. You could use a plain reverse SSH tunnel but most corporate firewalls will stop this.

The best solution is to install Apache2 and forward incoming HTTPS traffic to a local SSH server.

Spin up a new Ubuntu 16.04 EC2 instance with a public IP and connect to it with SSH. Be sure to allow HTTPS (port 443) traffic in from the public Internet. Also set up a public DNS name for your HTTPS certificates.

Install Apache2 and activate the modules you need for HTTPS and proxying.

sudo apt-get install apache2
sudo a2enmod ssl
sudo a2enmod proxy
sudo a2enmod proxy_http

You can provision certificates with LetsEncrypt or install your own.

Update your Apache2 configuration using the example below. Be sure to change the following lines to point to your SSL certificates.

SSLCertificateFile /etc/letsencrypt/live/your_site_name/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/your_site_name/privkey.pem

<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin webmaster@localhost

                DocumentRoot /var/www/html

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                #   SSL Engine Switch:
                #   Enable/Disable SSL for this virtual host.
                SSLEngine on
                SSLCertificateFile      /etc/letsencrypt/live/your_site_name/fullchain.pem
                SSLCertificateKeyFile /etc/letsencrypt/live/your_site_name/privkey.pem
                <FilesMatch "\.(cgi|shtml|phtml|php)$">
                                SSLOptions +StdEnvVars
                </FilesMatch>
                <Directory /usr/lib/cgi-bin>
                                SSLOptions +StdEnvVars
                </Directory>

        # proxytunnel
        ProxyRequests On
        AllowConnect 22
        <Proxy *>
            # Deny all proxying by default ...
            Require all denied
        </Proxy>
        <Proxy 127.0.0.1>
            # Now allow proxying by localhost only
            Require all granted
        </Proxy>

        </VirtualHost>
</IfModule>

There are lots of good guides on how to get this working if you get stuck. Just Google it: https://www.google.co.uk/search?q=ssh+over+https+apache2

2. Setup this script on your EC2 server

This script will automate the process of creating a new SSH key pair and finding a free local port to connect to. It will also trigger the SSM commands and setup the reverse tunnel for you.

Setup / configure awscli on your Ubuntu EC2 server and ensure it is up to date by running the following commands. Be sure to use an IAM service account for awscli configure that can access SSM to query instances and run commands.

sudo apt-get awscli
sudo pip3 install awscli --upgrade
sudo aws configure

Create a new local Linux user just for the SSH connection and remove their shell access. Be sure to set a complex password and harden your system accordingly. e.g. Install fail2ban

sudo adduser sshconnectuser
sudo usermod -s /bin/false sshconnectuser

Create a new script named sshconnect-server.sh with nano. Make it executable with chmod +x sshconnect-server.sh. Be sure to replace YOUR_PUBLIC_DNS_NAME_HERE with your EC2 instance public DNS name.

#!/bin/bash

# Configuration
SSH_USERNAME=sshconnectuser
SSH_HOST=YOUR_PUBLIC_DNS_NAME_HERE

# SSM document name
SSM_DOCUMENT="sshconnect-ssm"

# remove any old keys
sudo rm -f /dev/shm/id_rsa.pub
sudo rm -f /dev/shm/id_rsa

# generate a fresh key pair without a passphrase as sshconnectuser
sudo -H -u sshconnectuser bash -c "ssh-keygen -N '' -b 2048 -f /dev/shm/id_rsa"

# copy the key to the correct location for the sshconnect user
sudo mkdir -p /home/sshconnectuser/.ssh/
sudo cp /dev/shm/id_rsa.pub /home/sshconnectuser/.ssh/authorized_keys

# sshconnectuser should own the file
sudo chown sshconnectuser:sshconnectuser /home/sshconnectuser/.ssh/authorized_keys

# Copy the private key to a variable
SSH_KEY=$(sudo cat /dev/shm/id_rsa)

# Obtain a list of SSM agents
INSTANCE_LIST=$(aws ssm describe-instance-information --query "InstanceInformationList[*].[InstanceId, ComputerName]" --output text)

# Select open port on localhost starting with 11111
SSH_PORT=11111
while true; do
  nc -zv 127.0.0.1 ${SSH_PORT}
  if [ $? == 0 ]
  then
    SSH_PORT=$(expr ${SSH_PORT} + 1)
  else
    break
  fi
done

# Show a list of SSM instances to remotely access
INSTANCE=$(whiptail --title "Choose a host" --menu "Instance ID, ComputerName" 24 50 14 ${INSTANCE_LIST} 3>&1 1>&2 2>&3)

# Exit if menu canceled
if [[ ${?} == 1 ]]
then
  exit 0
fi

# Run the SSM task to trigger the remote host's reverse tunnel, this references the SSM document on AWS and passes in the instance / port / key parameters
aws ssm send-command --document-name ${SSM_DOCUMENT} --targets "Key=instanceids,Values=${INSTANCE}" \
--parameters "SSHHOST=${SSH_HOST},SSHPORTREMOTE=${SSH_PORT},SSHKEY=${SSH_KEY},SSHUSERNAME=${SSH_USERNAME}"

echo "Waiting 15 seconds for the connection"

sleep 15

# Connect to SSH reverse tunnel
ssh 127.0.0.1 -p ${SSH_PORT}

3. Create a new YAML AWS SSM document

Be sure to name it sshconnect-ssm so the script on your Ec2 instance can call it. More info about how to create a new SSM document: https://docs.aws.amazon.com/systems-manager/latest/userguide/create-ssm-doc.html

This document will take the variables from your sshconnect-server script and execute the correct commands on the remote host.

---
schemaVersion: "2.2"
description: "Copy a private key to a remote host and establish a reverse tunnel on a specific port.  Document on AWS must be named: sshconnect-ssm"
parameters:
  SSHKEY:
    type: "String"
    description: "SSH private key"
    default: ""
  SSHHOST:
    type: "String"
    description: "SSH server public DNS name"
    default: ""
  SSHUSERNAME:
    type: "String"
    description: "SSH username"
    default: "sshconnectuser"
  SSHPORTREMOTE:
    type: "String"
    description: "Remote port for SSH"
    default: ""
mainSteps:
- action: "aws:runShellScript"
  name: "keygen"
  inputs:
    runCommand:
      - "rm -f /dev/shm/id_rsa"
      - "touch /dev/shm/id_rsa"
      - "chmod 400 /dev/shm/id_rsa"
      - "echo '' > /dev/shm/id_rsa"
      - "/usr/local/bin/sshconnect-client.sh   "
      - "rm -f /dev/shm/id_rsa"

4. Setup SSM on your remote host

This page explains how to deploy the agent. Be sure to register it after you deploy it and confirm it is checking in.

Agent install for Ubuntu: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-manual-agent-install.html#agent-install-ubuntu-deb

Proxy configuration: https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-proxy-with-ssm-agent.html

5. Install ProxyTunnel and deploy this script on the remote machine

Create the script in /usr/local/bin/ and name it sshconnect-client.sh. Ensure you make it executable with sudo chmod +x /usr/local/bin/sshconnect-client.sh. This script will handle the connection back to the EC2 server. It will also attempt to work out the local proxy if one is being used.

#!/bin/bash

# This script should reside on the client system, it is called remotely by AWS SSM and will connect back over HTTPS with SSH and setup a reverse tunnel

SSH_HOST=${1}
SSH_USERNAME=${2}
SSH_PORT_REMOTE=${3}

TUNNEL_COMMAND_BASE="ProxyCommand proxytunnel -q -d 127.0.0.1:22"

# Work out the correct proxy scenario

# Is a local proxy between the HTTPS server and the client?
if $(grep -q 'https_proxy' /etc/environment)
then
  # Find proxy string and remove prefix
  PROXY_STRING=$(grep https_proxy /etc/environment | sed 's/"//g' | cut -d "=" -f 2 | sed 's/^........//')

  # Add proxy auth to the tunnel command if required
  if [[ ${PROXY_STRING} = *"@"* ]]
  then
    # Connect to authenticated proxy and then create an encrypted tunnel to HTTPS server
    # Environment variables must be used to support complex URL encoded passwords

    # Get only auth string
    PROXY_AUTH=$(echo ${PROXY_STRING} | cut -d "@" -f 1)

    # Get username
    set PROXYUSER=$(echo ${PROXY_AUTH} | cut -d ":" -f 1)

    # Get password
    set PROXYPASS=$(echo ${PROXY_AUTH} | cut -d ":" -f 2)

    PROXY_HOST=$(echo ${PROXY_STRING} | cut -d "@" -f 2)
  else
    PROXY_HOST=${PROXY_STRING}
  fi

  # Connect to local proxy
  TUNNEL_COMMAND_SUFFIX="-X -r ${SSH_HOST}:443 -p ${PROXY_HOST}"
else
 # No proxy in between
 TUNNEL_COMMAND_SUFFIX="-E -p ${SSH_HOST}:443"
fi

# Build tunnel command for correct scenario
TUNNEL_COMMAND="${TUNNEL_COMMAND_BASE} ${TUNNEL_COMMAND_SUFFIX}"

# Build the command
SSH_COMMAND="ssh -f -q -v -N \
-i /dev/shm/id_rsa \
-o 'UserKnownHostsFile=/dev/null' \
-o 'HostName ${SSH_HOST}' \
-o 'Port 443' \
-o 'StrictHostKeyChecking=no' \
-o '${TUNNEL_COMMAND}' \
-R${SSH_PORT_REMOTE}:localhost:22 ${SSH_USERNAME}@${SSH_HOST}"

# Run the built command
eval ${SSH_COMMAND}

6. Run the server side script

Call your sshconnect-server.sh script and it will prompt you for available SSM instances. Then select one that you deployed the client script to, it should connect back to your server over HTTPS and use the local proxy if one is required. You can then login with the username and password of the remote system.

Deployment
AWS
Ssh
Ssh Tunnel
Apache

--

--

Glen Tomkowiak

Written by Glen Tomkowiak

13 Followers

Things that interest me: cloud computing, cyber security, DevOps, and mobile / web development.

Help

Status

About

Careers

Blog

Privacy

Terms

Text to speech

Teams