Transferring Files Over SSH Using a Non-Standard Port: An In-Depth Practical Guide

The Secure Copy Protocol (SCP) is a ubiquitous method for securely transferring files between hosts over an SSH connection. By convention, SCP utilizes port 22 for establishing SSH connections. However, obscuring the SSH port can be beneficial for security and avoiding conflicts.

As an experienced DevOps architect and Linux systems engineer, I have helped numerous enterprises enhance their system security and compliance through SSH port obfuscation. This in-depth practical guide draws on my expertise securing mission-critical file transfers for Fortune 500 clients.

We will unpack:

• Standard vs non-standard SSH ports from a security perspective
• Methods for selecting obscure ports programmatically
• Step-by-step guidance on employing non-standard ports
• Log analysis for diagnosing connection issues
• Performance optimization considerations
• Failure scenarios and troubleshooting techniques
• Industry best practices for secure architecture

Whether you are aiming to boost security, meet complex compliance requirements, or avoid port conflicts, this comprehensive walkthrough has you covered.

SSH Port Visibility: Security Implications

The standard SSH port of 22 is widely recognized on the public internet. Attackers and opportunistic bots actively scan networks for open 22/TCP to identify systems possibly vulnerable to compromise. This table compares visibility between standard vs non-standard SSH ports:

Port Type	Visibility	Security Implications
Standard (22)	High Visibility	Frequent scanning and brute force attacks
Non-Standard	Low Visibility	Avoid most opportunistic threats

Obscuring SSH away from port 22 significantly reduces visibility and subsequently lowers the frequency of authorized access attempts. But it does not reduce the need for comprehensive security precautions like fail2ban, key-based authentication, etc. Think of it as "security through obscurity" – useful but not sufficient alone.

Choosing Appropriately Obscure Ports

While any port technically works for SSH, thoughtfully choosing your non-standard port offers maximum security benefits.

Certain port ranges are more heavily scanned than others based on common conventions and known services. For example, ports below 1024 are not ideal choices as they require superuser privileges and are easily discovered.

meanwhile, ports in the 49152 to 65535 range tend to see lower rates of unauthorized probing compared to other standard service port ranges:

Image Source: Tenable Research

Leveraging an intentionally obscure, randomized high-number port in this range combines the benefits of security through obscurity with true confidentiality for SSH connectivity.

One secure approach professionals use is programmatically generating a high-range random port on each server launch. This consistently shifts the target for bad actors without ongoing manual effort:

# Generate obscure port  
SSH_PORT=$(shuf -i49152-65535 -n1)

# Run SSH on obscure port
/usr/sbin/sshd -p ${SSH_PORT}

Now that we have covered how to intelligently select appropriately obscure ports, let‘s walk through applying this in practice across real-world use cases.

Step-by-Step Guide: Transferring Files via SCP with a Non-Standard SSH Port

The Secure Copy Protocol (SCP) is one of the most ubiquitous methods for trusted server-to-server and admin-to-machine file transfers. The -P option within SCP allows specifying non-standard SSH ports:

scp -P <port> <other options> 

Let‘s transfer important files between servers using SCP with SSH listening on port 54892 instead of 22:

1. Configure SSH on Source Server

As superuser, edit SSH daemon config file:

sudo vim /etc/ssh/sshd_config

Change or add the following, using your preferred non-standard port:

Port 54892

Save changes and restart SSH daemon:

sudo systemctl restart sshd

2. Configure SSH on Destination Server

Repeat the same sshd_config changes on the destination server:

Port 54892 

Restart SSH daemon there as well.

3. Update Firewalls on Source and Destination

On both systems, permit access for your selected port with UFW:

sudo ufw allow 54892/tcp

4. Initiate SCP Transfer

With SSH listening on 54892 on both systems, utilize SCP with -P port flag:

scp -P 54892 /path/to/file user@destination:/path/to/copy

This will securely copy /path/to/file from the source system to destination via SSH on port 54892.

5. Validate Functionality

Log into both servers via SSH on chosen port and confirm file presence as expected:

ssh -p 54892 user@server
ls -l /path/to/copy

By following this sequence, you now have confidential SCP file transfers protected by an obscured, non-standard SSH port.

Logging and Analysis for Diagnosing Connectivity Issues

If you encounter permission errors or connection issues using SCP with a custom SSH port, one of the first steps should be carefully analyzing relevant log files.

Debugging based solely on exit codes can lead down dead ends without sufficient context. Thorough log analysis provides visibility that is otherwise opaque.

Log Locations

SSH Daemon Logs:

• Ubuntu/Debian – /var/log/auth.log
• RHEL/CentOS – /var/log/secure
• Many other distros: /var/log/sshd.log

SCP Utility Logs:

• /var/log/scp.log
• Within system/user ~/.ssh/logs

Parse Logs for:

• Successful / failed connection attempts
• Errors establishing SSH sessions
• Authentication failures
• File transfer specifics like speed and volume
• Exact times of activities

For example, excessive authentication errors would indicate a password or permissions issue rather than a port conflict. Slow transfer speeds could suggest network bottlenecks rather than port configuration problems.

Analyzing logs provides invaluable troubleshooting context around if/where/when connections are actually failing when relying on non-standard SSH ports.

Performance Considerations for Production SCP at Scale

While moving to non-standard SSH ports provides immense security advantages, we must also consider performance impact when handling large volumes of SCP file transfers across several systems.

Every established SSH connection utilizes server memory, ephemeral ports, and other resources which could bottleneck bandwidth. Opening thousands of SCP connections across a high-traffic pipeline could inadvertently create capacity issues.

Recommended Optimization Strategies

• Increase sshd connection limits: Raise max startups and max sessions to avoid refused connections when overloaded.
• Enable SSH connection multiplexing: Leverage ControlMaster with MaxSessions to reuse connections efficiently.
• Distribute keys for agentless SCP: Remove expensive per-hop public key auth.
• Load balance SCP workers: Spread transfers across multiple servers.
• Tune TCP settings: Adjust window scaling, buffers, etc to maximize throughput.
• Profile memory usage: Spot check SSH/SCP memory consumption during transfers to anticipate scaling limits.

With careful architecture decisions and capacity planning around SSHD configuration, even large-volume SCP pipelines can thrive using non-standard SSH port security.

Failover Scenarios and Troubleshooting Techniques

While rare, there are some edge cases where utilizing a non-standard SSH port can introduce issues compared to relying on the common port 22 – if not planned for properly. Experienced DevOps engineers anticipate and design for these potential failure scenarios.

Common Failure Scenarios

• Network misconfiguration blocking the non-standard port
• Security policies explicitly denying obscure ports
• Hardcoded credential scripts break with non-22 ports
• Upstream NAT or proxies only pass traffic on standard ports

Troubleshooting Techniques

• Temporarily revert SSH port to 22 as rollback
• Place SSH on dual ports during transition period
• Create iptables rules to redirect old port 22 to new secure port
• Update coding artifacts like Ansible playbooks with new port
• Engage security teams to add exceptions for approved non-standard ports

A pragmatic troubleshooting methodology coupled with proactive mitigations around change management and legacy procedures enables realizing the benefits of non-standard SSH ports at scale with minimal risk.

Industry Best Practices for Secure Architecture

While this guide has covered techniques for obscuring SSH connectivity, properly implementing security in depth is critical as well. Port obfuscation complements rather than replaces other critical controls.

The Center for Internet Security (CIS) curates globally recognized security best practices and benchmarks. Their consensus recommendations to securely implement non-standard SSH ports include:

• Require strong passphrase authentication
• Install latest OS and SSH patches
• Limit access to authorized subnets
• Create individual system accounts
• Send logs to central SIEM system
• Enable automatic account lockouts
• Rotate keys and credentials periodically
• Perform regular vulnerability scans
• Maintain secure system baselines

No single change provides complete protection, but collectively implementing controls at the user, network, system, application, and physical layers substantially raises threat barriers.

Key Takeaways and Next Steps

We have covered a wealth of practical techniques and expert-level considerations when utilizing SCP and SSH connectivity over non-standard ports:

Key Takeaways

• Non-standard SSH ports significantly increase obscurity and reduce opportunistic threats – though other layered controls remain critical.
• Carefully choosing appropriately obscure ports based on visibility and range avoids pitfalls compared to haphazard ports.
• Updating SSHD configs, firewall rules, and SCP port flags allows quick transition to non-standard connectivity.
• Log analysis provides vital troubleshooting context around connection issues or failures.
• Load testing and optimization is key to reach 10x scale securely.
• Techniques like dual ports, coding updates, and security policy exceptions ease migrations.

Adopting these approaches allows organizations to meet rigorous enterprise security and compliance requirements without disrupting trusted pipelines for confidential data transfer.

If you found this guide useful or have additional questions, please reach out to continue the conversation! I welcome the chance to help strategize securing critical file movement workflows.