File Transfer Mastery

0.1 The Story: The Great Migration

Picture this: You have 2.5 million research files totaling 850GB scattered across various Windows file shares. Your deadline is approaching, and you need to transfer all this data to your Unix HPC cluster for analysis. Using Windows Explorer would take weeks and probably fail halfway through. Your research depends on getting this right.

This is where mastering file transfer becomes your superpower. Today, you’ll learn the professional tools that can handle massive data migrations reliably, efficiently, and with the ability to resume if something goes wrong.

0.2 The File Transfer Hierarchy of Needs

Just like Maslow’s hierarchy, file transfer has levels of sophistication:

1. Basic survival: cp (copy within same system)
2. Remote connection: scp (secure copy between systems)
   
3. Intelligent synchronization: rsync (the Swiss Army knife)
4. Parallel processing: Multiple streams and advanced techniques
5. Professional automation: Scripts that handle everything

We’ll climb this pyramid together, starting with the basics and reaching professional-level mastery.

0.3 Level 1: Local File Operations

Before we transfer files between systems, let’s master local operations:

0.3.1 The Basic Copy Command

cp source_file destination_file         # Copy single file
cp file1 file2 file3 destination_dir/   # Copy multiple files
cp -r source_dir destination_dir        # Copy entire directory

Real-world analogy: It’s like photocopying documents – you make an exact duplicate somewhere else.

0.3.2 The Move Command

mv old_name new_name                     # Rename file
mv file1 file2 destination_dir/         # Move files to directory
mv source_dir destination_dir           # Move entire directory

Real-world analogy: It’s like physically moving boxes from one room to another – the original disappears from the old location.

0.3.3 Essential Copy Options

cp -v source dest        # Verbose (show what's being copied)
cp -p source dest        # Preserve timestamps and permissions
cp -u source dest        # Update only (copy if newer)
cp -i source dest        # Interactive (ask before overwriting)

0.4 Level 2: Remote File Transfer with SCP

scp (Secure Copy Protocol) is your first tool for transferring files between different systems:

0.4.1 Basic SCP Syntax

scp local_file user@remote_host:/remote/path/
scp user@remote_host:/remote/file local_path/
scp -r local_directory user@remote_host:/remote/path/

Real-world analogy: It’s like using a secure courier service to send packages between cities.

0.4.2 Practical HPC Examples

# Upload data to HPC cluster
scp experiment_data.csv username@hpc.university.edu:/home/username/data/

# Download results from HPC
scp username@hpc.university.edu:/results/analysis.out ./results/

# Upload entire project directory
scp -r project_folder username@hpc.university.edu:/scratch/username/

0.4.3 SCP Best Practices

scp -p file user@host:/path/        # Preserve file attributes
scp -C file user@host:/path/        # Compress during transfer
scp -v file user@host:/path/        # Verbose output for debugging

0.5 Level 3: The Master Tool - rsync

rsync is the professional’s choice for file transfer. It’s like having a smart assistant who only copies what’s necessary:

0.5.1 Why rsync is Superior

• Incremental: Only transfers changed parts of files
• Resumable: Can continue interrupted transfers
• Verification: Checks data integrity
• Flexible: Tons of options for every scenario

0.5.2 Basic rsync Syntax

rsync source destination                    # Basic sync
rsync -av source destination               # Archive mode with verbose
rsync -av source/ destination/             # Sync contents (note the trailing slash)
rsync -av source user@host:/destination/   # Remote sync

Real-world analogy: It’s like having a brilliant moving company that remembers what they’ve already moved and only brings the new or changed items.

0.6 Essential rsync Options (The Power Combo)

rsync -avz source destination

Let’s break this down: - -a (archive): Preserves permissions, timestamps, symbolic links - -v (verbose): Shows what’s being transferred - -z (compress): Compresses data during transfer

Real-world analogy: It’s like having a professional moving service that: - Handles everything carefully (archive) - Gives you a detailed receipt (verbose)
- Packs efficiently to save space (compress)

0.7 Advanced rsync for HPC Scenarios

0.7.1 The Complete Professional Command

rsync -avz --progress --partial --stats source user@host:/destination/

Additional options explained: - –progress: Shows transfer progress in real-time - –partial: Keeps partially transferred files (for resuming) - –stats: Shows detailed transfer statistics

0.7.2 Handling Millions of Files

# For massive transfers, add these options
rsync -avz --progress --partial --stats --human-readable \
  --itemize-changes source user@host:/destination/

• –human-readable: Shows sizes in KB, MB, GB
• –itemize-changes: Shows exactly what changed for each file

0.7.3 Excluding Unwanted Files

# Exclude temporary and system files
rsync -avz --exclude='*.tmp' --exclude='*.log' --exclude='.DS_Store' \
  source user@host:/destination/

# Use exclude file for complex patterns
echo "*.tmp" > exclude_list.txt
echo "*.log" >> exclude_list.txt
rsync -avz --exclude-from=exclude_list.txt source user@host:/destination/

0.8 Real-World HPC Transfer Scenarios

0.8.1 Scenario 1: Initial Data Upload (The Big Bang)

# First-time upload of massive dataset
rsync -avz --progress --stats --human-readable \
  /mnt/windows_share/research_data/ \
  username@hpc.university.edu:/scratch/username/data/

# This command:
# - Transfers everything efficiently
# - Shows progress so you don't panic
# - Gives statistics at the end
# - Uses human-readable sizes

0.8.2 Scenario 2: Daily Synchronization

# Daily sync of new/changed files only
rsync -avz --progress --update --stats \
  /mnt/windows_share/research_data/ \
  username@hpc.university.edu:/scratch/username/data/

# The --update flag only copies files that are newer

0.8.3 Scenario 3: Selective Transfer

# Transfer only specific file types from specific time period
rsync -avz --progress --include='*.csv' --include='*.txt' \
  --exclude='*' --newer-than='2023-12-01' \
  /mnt/windows_share/research_data/ \
  username@hpc.university.edu:/scratch/username/data/

0.8.4 Scenario 4: Resuming Interrupted Transfers

# If transfer was interrupted, just run the same command again
rsync -avz --progress --partial --stats \
  /mnt/windows_share/research_data/ \
  username@hpc.university.edu:/scratch/username/data/

# rsync will figure out what's missing and continue

0.9 Monitoring Large Transfers

0.9.1 Real-time Monitoring

# In another terminal, monitor the destination
watch -n 5 'du -sh /scratch/username/data && ls -la /scratch/username/data | wc -l'

# This shows:
# - Total size transferred so far
# - Number of files transferred
# - Updates every 5 seconds

0.9.2 Transfer Speed Optimization

# Limit bandwidth (useful during business hours)
rsync -avz --progress --bwlimit=10000 source user@host:/dest/  # 10MB/s limit

# Multiple parallel streams (for very large transfers)
# We'll cover this in the parallel processing post

0.10 Windows to Unix: Special Considerations

0.10.1 Handling Windows File Shares

# First, mount the Windows share (on Linux)
sudo mkdir /mnt/research_data
sudo mount -t cifs //windows-server/share /mnt/research_data \
  -o username=your_windows_user,uid=$(id -u),gid=$(id -g)

# Then transfer
rsync -avz --progress /mnt/research_data/ username@hpc:/scratch/data/

0.10.2 Character Encoding Issues

# Handle special characters in filenames
rsync -avz --iconv=utf-8,utf-8 source destination

0.10.3 Permission Fixes

# Fix permissions after transfer from Windows
find /scratch/username/data -type f -exec chmod 644 {} \;  # Files
find /scratch/username/data -type d -exec chmod 755 {} \;  # Directories

0.11 Professional Transfer Scripts

0.11.1 The Bulletproof Transfer Script

#!/bin/bash
# transfer_data.sh - Professional data transfer script

SOURCE="/mnt/windows_share/research_data/"
DEST="username@hpc.university.edu:/scratch/username/data/"
LOG_FILE="transfer_$(date +%Y%m%d_%H%M%S).log"

echo "Starting transfer at $(date)" | tee -a $LOG_FILE

# The main transfer with all safety features
rsync -avz --progress --partial --stats --human-readable \
  --log-file=$LOG_FILE \
  --exclude='*.tmp' --exclude='*.log' --exclude='.DS_Store' \
  "$SOURCE" "$DEST" 2>&1 | tee -a $LOG_FILE

RESULT=$?

if [ $RESULT -eq 0 ]; then
    echo "Transfer completed successfully at $(date)" | tee -a $LOG_FILE
else
    echo "Transfer failed with exit code $RESULT at $(date)" | tee -a $LOG_FILE
    echo "Check $LOG_FILE for details"
fi

0.11.2 Usage

chmod +x transfer_data.sh
./transfer_data.sh

0.12 Verification and Integrity Checking

0.12.1 Verify Transfer Integrity

# Compare file counts
echo "Source files:"
find /mnt/windows_share/research_data -type f | wc -l

echo "Destination files:"
ssh username@hpc.university.edu 'find /scratch/username/data -type f | wc -l'

# Compare total sizes
echo "Source size:"
du -sh /mnt/windows_share/research_data

echo "Destination size:"
ssh username@hpc.university.edu 'du -sh /scratch/username/data'

0.12.2 Checksum Verification (for critical data)

# Generate checksums on source
find /mnt/windows_share/research_data -type f -exec md5sum {} \; > source_checksums.txt

# Generate checksums on destination
ssh username@hpc.university.edu \
  'find /scratch/username/data -type f -exec md5sum {} \;' > dest_checksums.txt

# Compare (this requires some processing to handle path differences)

0.13 Common Transfer Problems and Solutions

0.13.1 Problem 1: Transfer Keeps Failing

Symptoms: Connection drops, partial transfers Solution: Use rsync with --partial and --timeout options

rsync -avz --progress --partial --timeout=300 source dest

0.13.2 Problem 2: Too Slow

Symptoms: Transfer taking forever Solutions:

# Use compression
rsync -avz source dest

# Limit what you transfer
rsync -avz --exclude='*.tmp' source dest

# Use multiple streams (advanced)
# We'll cover this in parallel processing

0.13.3 Problem 3: Permission Denied

Symptoms: Cannot write to destination Solution: Check SSH keys and destination permissions

# Test SSH connection first
ssh username@hpc.university.edu 'ls -la /scratch/username/'

0.13.4 Problem 4: Running Out of Space

Symptoms: “No space left on device” Solution: Check space before and during transfer

# Check destination space
ssh username@hpc.university.edu 'df -h /scratch'

# Monitor during transfer
watch -n 30 "ssh username@hpc.university.edu 'df -h /scratch'"

0.14 Quick Reference: Transfer Command Cheat Sheet

Task	Command	Use Case
Local copy	cp -r source dest	Same system
Remote copy	scp -r source user@host:/dest	Simple remote transfer
Smart sync	rsync -avz source user@host:/dest	Professional transfers
Resume transfer	rsync -avz --partial source dest	Continue interrupted
Exclude files	rsync -avz --exclude='*.tmp' source dest	Skip unwanted files
Bandwidth limit	rsync --bwlimit=5000 source dest	Limit network usage
Progress monitor	rsync -avz --progress source dest	See transfer progress

0.15 Practice Exercises

ImportantYour Transfer Training Mission

1. Create test data structure

   mkdir -p test_transfer/{data,results,logs}
   echo "Test file 1" > test_transfer/data/file1.txt
   echo "Test file 2" > test_transfer/data/file2.txt
   echo "Log entry" > test_transfer/logs/app.log

2. Practice local rsync

   rsync -avz --progress test_transfer/ test_transfer_copy/

3. Test exclusion patterns

   rsync -avz --exclude='*.log' test_transfer/ test_transfer_no_logs/

4. Simulate resume capability

   # Start a transfer and interrupt it (Ctrl+C)
   rsync -avz --progress --partial large_file destination/
   # Then run the same command again to resume

0.16 What’s Next?

Now that you’re a file transfer master, you’re ready to learn about Environment Mastery. This will teach you how to run transfers in the background while you work on other tasks – essential for managing long-running HPC operations.

Remember: Efficient file transfer is often the bottleneck in HPC workflows. Master these tools, and you’ll save hours of waiting and frustration. Your research will move faster, and you’ll sleep better knowing your transfers are reliable and resumable.

TipTransfer Mastery Challenge

Before moving on, try these real-world scenarios: 1. Transfer a directory with mixed file types, excluding temporary files 2. Start a large transfer, interrupt it, then resume it 3. Transfer files while limiting bandwidth to 1MB/s 4. Compare transfer times between scp and rsync for the same data

This practice will prepare you for real HPC data migration challenges!

---

You’ve mastered the art of moving data efficiently across systems. Next, we’ll learn how to manage these transfers in the background while you focus on other important work!