SSH and Remote Access

0.1 The Story: The Digital Teleporter

Imagine you’re a researcher who needs to access powerful supercomputers located in data centers around the world. These machines have thousands of CPU cores and terabytes of memory, but they’re physically hundreds of miles away. You need a way to teleport your consciousness into these remote machines, work on them as if you were sitting right there, and do it all with bank-level security.

This isn’t science fiction – it’s SSH (Secure Shell), the digital teleporter that has connected researchers to remote systems for decades. With SSH, your laptop becomes a window into any Unix system on the planet, allowing you to run experiments, transfer files, and manage data across continents as easily as working on your local machine.

Today, you’ll master the art of secure remote access, becoming a digital nomad who can work anywhere, anytime, on any system.

0.2 Understanding SSH: Your Digital Passport

SSH is like having a secure, encrypted tunnel between your computer and a remote system. Everything you type is encrypted before leaving your machine, travels securely across the internet, and is decrypted only when it reaches the destination.

Real-world analogy: SSH is like having a private, armored car with tinted windows that can instantly transport you to any building in the world. Nobody can see what you’re doing inside, and you arrive safely at your destination.

0.2.1 The SSH Architecture

Your Computer  ←→  Internet  ←→  Remote HPC System
    (Client)    [Encrypted]      (Server)

0.3 Basic SSH Connection

0.3.1 Your First Remote Connection

# Basic SSH connection
ssh username@remote-server.edu

# SSH with specific port (if not default 22)
ssh -p 2222 username@remote-server.edu

# SSH with verbose output (for troubleshooting)
ssh -v username@remote-server.edu

0.3.2 What Happens During Connection

1. Authentication: You prove who you are (password or key)
2. Encryption: A secure tunnel is established
3. Shell: You get a command prompt on the remote system
4. Work: Everything you type runs on the remote machine

Real-world analogy: It’s like calling someone on the phone, but instead of just talking, you can actually control their computer as if you were sitting at their desk.

0.4 SSH Key Authentication: The Master Key System

Password authentication is like having to show your ID every time you enter a building. SSH keys are like having a master key that automatically opens the door.

0.4.1 Generating Your SSH Key Pair

# Generate a new SSH key pair
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

# Generate with specific filename
ssh-keygen -t rsa -b 4096 -f ~/.ssh/hpc_cluster_key

# Generate Ed25519 key (more modern, shorter)
ssh-keygen -t ed25519 -C "your_email@example.com"

This creates two files: - Private key (~/.ssh/id_rsa): Keep this secret! Never share it. - Public key (~/.ssh/id_rsa.pub): This is safe to share.

Real-world analogy: The private key is like your house key (keep it safe), and the public key is like giving the locksmith your lock’s specifications so they can verify your key works.

0.4.2 Installing Your Public Key on Remote Systems

# Copy your public key to remote server
ssh-copy-id username@remote-server.edu

# Manual method if ssh-copy-id isn't available
cat ~/.ssh/id_rsa.pub | ssh username@remote-server.edu "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized_keys"

# Verify the key was installed
ssh username@remote-server.edu "cat ~/.ssh/authorized_keys"

0.4.3 Testing Key-Based Authentication

# This should now work without asking for password
ssh username@remote-server.edu

# If it still asks for password, check permissions:
ssh username@remote-server.edu "chmod 700 ~/.ssh && chmod 600 ~/.ssh/authorized_keys"

0.5 SSH Configuration: Your Connection Management System

Instead of typing long SSH commands every time, create a configuration file that remembers your settings.

0.5.1 Creating SSH Config File

# Create/edit SSH config
nano ~/.ssh/config

0.5.2 Sample SSH Configuration

# ~/.ssh/config

# HPC Cluster
Host hpc
    HostName cluster.university.edu
    User your_username
    Port 22
    IdentityFile ~/.ssh/hpc_cluster_key
    ServerAliveInterval 60
    ServerAliveCountMax 3

# Development Server
Host devserver
    HostName dev.company.com
    User developer
    Port 2222
    IdentityFile ~/.ssh/dev_key
    ForwardX11 yes

# Jump Host Configuration (for servers behind firewalls)
Host internal-server
    HostName 192.168.1.100
    User researcher
    ProxyJump jumphost.university.edu

# Default settings for all hosts
Host *
    ServerAliveInterval 60
    ServerAliveCountMax 3
    Compression yes

0.5.3 Using SSH Config

# Now you can connect with simple names
ssh hpc              # Connects to cluster.university.edu
ssh devserver        # Connects to dev.company.com
ssh internal-server  # Connects through jump host

Real-world analogy: SSH config is like having speed dial for phone numbers – instead of remembering long numbers, you just press one button.

0.6 Advanced SSH Techniques

0.6.1 SSH Tunneling: Creating Secure Pathways

SSH can create secure tunnels for various purposes:

# Local port forwarding (access remote service locally)
ssh -L 8080:localhost:80 username@remote-server.edu
# Now http://localhost:8080 on your machine connects to port 80 on remote server

# Remote port forwarding (make local service available remotely)
ssh -R 9090:localhost:3000 username@remote-server.edu
# Now port 9090 on remote server connects to port 3000 on your machine

# Dynamic port forwarding (SOCKS proxy)
ssh -D 1080 username@remote-server.edu
# Use localhost:1080 as SOCKS proxy in your browser

0.6.2 X11 Forwarding: Running Remote GUI Applications

# Enable X11 forwarding
ssh -X username@remote-server.edu

# Or in SSH config:
Host hpc
    ForwardX11 yes
    ForwardX11Trusted yes

# Now you can run GUI applications remotely
ssh hpc
firefox &          # Opens Firefox from remote server on your screen
matlab &           # Opens MATLAB GUI remotely

0.6.3 SSH Agent: Managing Multiple Keys

# Start SSH agent
eval $(ssh-agent)

# Add your keys to agent
ssh-add ~/.ssh/id_rsa
ssh-add ~/.ssh/hpc_cluster_key

# List loaded keys
ssh-add -l

# Remove all keys from agent
ssh-add -D

0.7 Real-World HPC SSH Scenarios

0.7.1 Scenario 1: Multi-Hop Connection to Secure Cluster

# ~/.ssh/config for complex HPC access

# Jump host (gateway server)
Host gateway
    HostName gateway.hpc-center.edu
    User your_username
    IdentityFile ~/.ssh/hpc_key
    ServerAliveInterval 30

# Compute cluster (behind firewall)
Host cluster
    HostName compute-cluster.internal
    User your_username
    ProxyJump gateway
    IdentityFile ~/.ssh/hpc_key
    ServerAliveInterval 30

# Specific compute node
Host node001
    HostName node001.compute-cluster.internal
    User your_username
    ProxyJump gateway
    IdentityFile ~/.ssh/hpc_key

# Usage
ssh cluster          # Connects through gateway automatically
ssh node001          # Connects to specific compute node

0.7.2 Scenario 2: Automated HPC Workflow Script

#!/bin/bash
# hpc_workflow.sh - Automated remote job submission and monitoring

HPC_HOST="cluster"
JOB_DIR="/scratch/username/current_job"
LOCAL_SCRIPTS="./scripts"

echo "Starting HPC workflow..."

# 1. Prepare remote environment
ssh $HPC_HOST "mkdir -p $JOB_DIR"

# 2. Transfer job scripts
scp -r $LOCAL_SCRIPTS/* $HPC_HOST:$JOB_DIR/

# 3. Submit job and capture job ID
JOB_ID=$(ssh $HPC_HOST "cd $JOB_DIR && bsub < job_script.sh" | grep -oP '(?<=Job <)\d+(?=>)')

echo "Job submitted with ID: $JOB_ID"

# 4. Monitor job status
while true; do
    STATUS=$(ssh $HPC_HOST "bjobs $JOB_ID 2>/dev/null | tail -n 1 | awk '{print \$3}'")
    
    if [ "$STATUS" = "DONE" ]; then
        echo "Job completed successfully!"
        break
    elif [ "$STATUS" = "EXIT" ]; then
        echo "Job failed!"
        ssh $HPC_HOST "cat $JOB_DIR/job_output.err"
        exit 1
    else
        echo "Job status: $STATUS"
        sleep 30
    fi
done

# 5. Retrieve results
mkdir -p results/
scp -r $HPC_HOST:$JOB_DIR/output/* results/

echo "Workflow completed. Results downloaded to ./results/"

0.7.3 Scenario 3: Interactive HPC Session with Screen/Tmux

#!/bin/bash
# hpc_interactive.sh - Start persistent interactive session

HPC_HOST="cluster"
SESSION_NAME="research_$(date +%Y%m%d)"

echo "Starting interactive HPC session..."

# Connect and start tmux session
ssh -t $HPC_HOST "tmux new-session -d -s $SESSION_NAME && tmux send-keys 'cd /scratch/\$USER' Enter && tmux attach -t $SESSION_NAME"

# Alternative with screen
# ssh -t $HPC_HOST "screen -S $SESSION_NAME"

0.8 SSH Security Best Practices

0.8.1 Hardening SSH Configuration

# ~/.ssh/config security settings
Host *
    # Use only key authentication
    PasswordAuthentication no
    PubkeyAuthentication yes
    
    # Use strong ciphers
    Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
    
    # Disable weak algorithms
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group16-sha512
    
    # Connection timeouts
    ServerAliveInterval 60
    ServerAliveCountMax 3
    ConnectTimeout 30
    
    # Disable X11 forwarding by default (enable per host if needed)
    ForwardX11 no

0.8.2 Managing SSH Keys Securely

# Set proper permissions on SSH directory
chmod 700 ~/.ssh
chmod 600 ~/.ssh/config
chmod 600 ~/.ssh/id_rsa          # Private key
chmod 644 ~/.ssh/id_rsa.pub      # Public key
chmod 600 ~/.ssh/authorized_keys
chmod 600 ~/.ssh/known_hosts

# Use passphrase-protected keys for sensitive systems
ssh-keygen -t ed25519 -f ~/.ssh/sensitive_system_key
# Enter strong passphrase when prompted

0.8.3 SSH Key Management Script

#!/bin/bash
# manage_ssh_keys.sh - Manage SSH keys across multiple systems

SYSTEMS=("hpc1.edu" "hpc2.edu" "cluster.org")
PUBLIC_KEY="$HOME/.ssh/id_rsa.pub"

if [ ! -f "$PUBLIC_KEY" ]; then
    echo "Generating new SSH key..."
    ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa
fi

echo "Installing SSH key on remote systems..."
for system in "${SYSTEMS[@]}"; do
    echo "Installing key on $system..."
    ssh-copy-id -i "$PUBLIC_KEY" "$system"
    
    if [ $? -eq 0 ]; then
        echo "✓ Key installed on $system"
    else
        echo "✗ Failed to install key on $system"
    fi
done

echo "Testing connections..."
for system in "${SYSTEMS[@]}"; do
    echo -n "Testing $system... "
    if ssh -o ConnectTimeout=5 -o BatchMode=yes "$system" "echo 'OK'" 2>/dev/null; then
        echo "✓ Connected"
    else
        echo "✗ Failed"
    fi
done

0.9 Troubleshooting SSH Connections

0.9.1 Common SSH Problems and Solutions

0.9.1.1 Problem 1: “Permission denied (publickey)”

# Check if key is loaded in SSH agent
ssh-add -l

# If not loaded, add it
ssh-add ~/.ssh/id_rsa

# Check remote server permissions
ssh username@server "ls -la ~/.ssh/"
# Should show:
# drwx------  ~/.ssh/
# -rw-------  ~/.ssh/authorized_keys

0.9.1.2 Problem 2: “Connection timed out”

# Test with verbose output
ssh -v username@server

# Try different port
ssh -p 2222 username@server

# Check if server is reachable
ping server-hostname
telnet server-hostname 22

0.9.1.3 Problem 3: “Host key verification failed”

# Remove old host key
ssh-keygen -R hostname

# Or edit known_hosts file
nano ~/.ssh/known_hosts
# Remove the line with the problematic hostname

0.9.2 SSH Connection Debugging

# Maximum verbosity for debugging
ssh -vvv username@server

# Test specific configuration
ssh -F /path/to/config username@server

# Ignore known_hosts (for testing only!)
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null username@server

0.10 SSH Performance Optimization

0.10.1 Speeding Up SSH Connections

# ~/.ssh/config optimizations
Host *
    # Reuse connections
    ControlMaster auto
    ControlPath ~/.ssh/sockets/%r@%h-%p
    ControlPersist 600
    
    # Enable compression
    Compression yes
    
    # Faster cipher (less secure, use only on trusted networks)
    # Cipher arcfour
    
    # Keep connections alive
    ServerAliveInterval 60
    ServerAliveCountMax 3

0.10.2 Connection Multiplexing Setup

# Create socket directory
mkdir -p ~/.ssh/sockets

# First connection establishes master
ssh hpc

# Subsequent connections reuse the master (much faster)
ssh hpc  # This will be instant!
scp file.txt hpc:~/  # This will also be faster

0.11 Quick Reference: SSH Command Cheat Sheet

Task	Command	Example
Basic connection	ssh user@host	ssh alice@hpc.edu
Connect with key	ssh -i keyfile user@host	ssh -i ~/.ssh/hpc_key alice@hpc.edu
Connect to port	ssh -p port user@host	ssh -p 2222 alice@hpc.edu
X11 forwarding	ssh -X user@host	ssh -X alice@hpc.edu
Local port forward	ssh -L local:remote user@host	ssh -L 8080:localhost:80 alice@hpc.edu
Generate key pair	ssh-keygen -t type	ssh-keygen -t rsa -b 4096
Copy public key	ssh-copy-id user@host	ssh-copy-id alice@hpc.edu
Add key to agent	ssh-add keyfile	ssh-add ~/.ssh/id_rsa

0.12 Practice Exercises

ImportantYour SSH Mastery Training

1. Set up key-based authentication

   # Generate SSH key pair
   # Install public key on a remote system
   # Test passwordless login

2. Create SSH configuration

   # Set up ~/.ssh/config with:
   # - Host aliases
   # - Different keys for different systems
   # - Connection optimization settings

3. Practice SSH tunneling

   # Set up port forwarding
   # Access a remote web service locally
   # Create a SOCKS proxy

4. Automate remote tasks

   # Write a script that:
   # - Connects to remote system
   # - Runs commands
   # - Transfers files
   # - Reports results

0.13 What’s Next?

Now that you’re an SSH master, you’re ready to learn about File Transfer Mastery. SSH is the foundation for secure file transfers, and you’ll build on these skills to move massive amounts of data efficiently between systems.

Remember: SSH is your gateway to the HPC world. Every data transfer, every remote job, every collaboration with distant systems starts with a secure SSH connection. Master these skills, and you’ll work seamlessly across any number of remote systems as if they were all sitting on your desk.

TipSSH Mastery Challenge

Before moving on, try this real-world scenario: 1. Set up passwordless access to a remote system 2. Create a complex SSH configuration with multiple hosts 3. Write a script that automates a multi-step remote workflow 4. Set up secure tunneling for a specific application

This mirrors the daily work of HPC researchers who manage multiple remote systems!

---

You’ve mastered the art of secure remote access – the ability to work anywhere in the digital universe. Next, we’ll learn to move your data across this universe with the speed and efficiency of a digital freight system!