BIPI
BIPI

Killing the Bastion Host: SSM Session Manager in Practice

Cloud Security

SSM Session Manager replaces bastion hosts with IAM-based access, session logging, and no inbound ports. Migration is mostly straightforward, but a few sharp edges trip up teams.

By Arjun Raghavan, Security & Systems Lead, BIPI · February 22, 2024 · 6 min read

#aws#ssm#bastion#access

The bastion host (sometimes called jump box) has been the standard pattern for SSH access to private EC2 instances for over a decade. A single hardened EC2 instance with port 22 open to a corp IP range, individual SSH keys per user, sometimes Duo MFA, and audit logs that nobody reads. We have replaced bastions in dozens of AWS accounts and the result is uniformly better: IAM-based access, no inbound ports, full session logging in S3 and CloudWatch, and port forwarding for database access that beats SSH tunneling.

What SSM Session Manager actually does

Session Manager is part of AWS Systems Manager. Every EC2 instance with the SSM Agent installed and an IAM instance profile that allows ssm:UpdateInstanceInformation can be connected to via Session Manager. The connection goes outbound from the instance to the SSM endpoints (which is why VPC endpoints for ssm, ssmmessages, and ec2messages matter). No inbound ports on the instance. No SSH keys to manage. No bastion to harden.

Access is gated by IAM. To start a session, the calling identity needs ssm:StartSession against the target instance ARN. To restrict who can connect where, write IAM policies that scope ssm:StartSession to specific tag values. To log sessions, configure SSM session preferences to ship session output to an S3 bucket and CloudWatch Logs group.

Setup requirements

For Session Manager to work, four things must be true:

  1. SSM Agent installed on the instance. Amazon Linux 2, Amazon Linux 2023, and recent Ubuntu AMIs have it preinstalled.
  2. Instance profile attached with AmazonSSMManagedInstanceCore policy (or equivalent custom permissions).
  3. Outbound network connectivity to SSM endpoints. Either via internet (NAT) or VPC endpoints (com.amazonaws.<region>.ssm, ssmmessages, ec2messages).
  4. SSM agent can reach the endpoints. Check with sudo amazon-ssm-agent (or systemctl status amazon-ssm-agent) and the agent logs at /var/log/amazon/ssm/.

The VPC endpoint requirement matters if you are running fully private subnets without NAT. In that case, all three SSM endpoints must be provisioned, and the endpoint security groups must allow HTTPS from the instances.

Port forwarding kills the SSH tunnel

Session Manager supports port forwarding, including forwarding to remote hosts. To access an RDS instance from a developer laptop:

aws ssm start-session --target i-0123 --document-name AWS-StartPortForwardingSessionToRemoteHost --parameters host=mydb.cluster-xyz.rds.amazonaws.com,portNumber=5432,localPortNumber=5432

That tunnels localhost:5432 on the laptop through the EC2 instance to the RDS endpoint. No SSH tunnel, no bastion, no inbound ports. The 'jump box' for database access becomes a small EC2 instance that exists only to be a network entry point, with no humans ever logging into it interactively.

Session logging

Session logging is configurable per AWS account through SSM session preferences. Configure the S3 bucket, CloudWatch log group, and KMS key. Every session's input and output is logged. We integrate this into the SIEM pipeline (typically EventBridge to a Lambda that ships sanitized session metadata to Splunk or Elastic).

Compared to bastion hosts where SSH logs depend on configuring sshd_config for verbose logging, capturing PTY output via auditd or a tool like teleport, and trusting that the bastion has not been tampered with, this is a huge improvement.

Migration friction

Three common issues during migration:

  • Tooling that hardcodes SSH (Ansible, custom deploy scripts). Ansible supports community.aws.ssm connection plugin but it is slower and less mature than SSH.
  • File transfer. scp does not work over Session Manager directly. Workarounds: aws ssm start-session with port forwarding to local sshd (if running), or aws s3 cp via the instance.
  • Long-running sessions. Session Manager sessions timeout after 20 minutes of inactivity by default. Configurable to up to 60 minutes. Long-running background work needs nohup or screen/tmux.

What this does not solve

SSM Session Manager is for accessing EC2 (and on-prem instances via SSM Agent). For Kubernetes, you still need kubectl with proper auth. For RDS, port forwarding works but is not the only option (RDS IAM authentication is better long-term). For ECS, ECS Exec is the equivalent feature (also SSM-based).

Killing the bastion is one step. Killing all long-lived human credentials in the AWS environment is the bigger project, and SSM is one of the simpler ones to start with.

Read more field notes, explore our services, or get in touch at info@bipi.in. Privacy Policy · Terms.