SSH

From Just another day in the life of a linux sysadmin
Jump to navigation Jump to search

What SSH is

SSH (stands for Secure SHell) is a protocol for Unix-like systems (Windows also supports it to some degree) by where you can connect to a remote server and administer it through a command line driven interface. It provides an encrypted channel for content traveling both ways, and is built to be solid. The CLI shell interfaces are based upon the standard shell technologies, and in the past have been handled by a tty (TeleTYpe), although more commonly they recently are a pty (PsudoTerminal device). A TTY denotes a terminal device that provides text based interaction and is backed by a hardware device, PTY is the same thing emulated in software.

OpenSSH distributes two major packages with their SSH installation - the OpenSSH server daemon, and the OpenSSH client.

Updating OpenSSH

OpenSSH should never be updated. OpenSSH is a base package paired to your version of the distribution running on a computer. Different versions are compatible with each other, and as long as you are on that distribution and the distribution does not change (very unlikely) your OpenSSH version will not change. Security issues with that version are ported back to that version by the distribution maintainer - this means that OpenSSH is kept secure. As such, installing OpenSSH outside of the repository removes the backporting of anything to the product, and and raises a risk of additional security vulnerabilities and failures.

Most security issues for not-EOL distributions can be covered by backported patches - patches for security fixes, ported to older versions of the service. Some others are covered by changing the configuration to the product - for instance disabling Protocol version 1 and only leaving Protocol version 2 active.

OpenSSH Server

SSH Port

When changing the port for SSH on a server, please ensure that the Auth tab for the subaccount is updated, which will update nagios on which port to monitor. The default OpenSSH port is port 22. This is the port SSH listens on by default. You can change this by editing the config file:

vim /etc/ssh/sshd_config

change

Port 22
Protocol 2

to

Port 25566
Protocol 2

Then, edit your firewall settings to allow the new port and remove the old one: (in this case CSF)

vim /etc/csf/csf.conf

Don't forget to change the auth settings in billing by going to Sub-accounts, selecting the server, then the "AUTH" tab and editing the settings to put the new port number in. It will then add that port to the SSH login string on the front page. The next step is to restart SSH and test out the new port. This will save you and your co-workers a lot of grief

OLD <source lang='text'>

# Allow incoming TCP ports
TCP_IN = "20,21,22,25,53,80,110,143,443,465,514,587,667,993,995,2077,2078,2082,2083,2086,2087,2095,2096,8000,8080,33000
# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,37,43,53,80,110,113,443,514,587,667,873,2087,2089,2703,8000,8080,33000"

</source> NEW <source lang='text'>

# Allow incoming TCP ports
TCP_IN = "20,21,25,53,80,110,143,443,465,514,587,667,993,995,2077,2078,2082,2083,2086,2087,2095,2096,8000,8080,25566,33000
# Allow outgoing TCP ports
TCP_OUT = "20,21,22,25,37,43,53,80,110,113,443,514,587,667,873,2087,2089,2703,8000,8080,33000"

</source>

In this case, I have removed port 22 and added port number 25566 for SSH to listen on for incoming requests. Note that the outgoing port should not change. Next, restart both your firewall...

csf -r 

or

apf -r 

and the SSH service on the server.

{{#lst:OpenSSH|restart_ssh}} Since SSH kicks off subprocesses every time a connection is made, your current session isn't affected by this, but if you SSH in again, you can test this.

To test for the new listening port, run:

netstat -plunt | grep ssh
tcp        0      0 0.0.0.0:25566               0.0.0.0:*                   LISTEN      6993/sshd           
tcp        0      0 :::25566                    :::*                        LISTEN      6993/sshd

Disable Root Login


When changing the user for SSH on a server, please ensure that the Auth tab for the subaccount is updated, so that other techs know which user to use to connect to the machine. This will change the copy-pasteable login info with the new user on the main billing page. Please also ensure that the root escalation method is noted, so that we can manage the server effectively. If you just disable root logins, you (more than likely) won't have a user that can escalate to root. First, we'll need to create a user. Typically the user is called sshuser, but you can choose whatever you'd like:

useradd $USER_NAME

Now to set a password:

passwd $USER_NAME

To actually allow this user to escalate to root, you need to add it to the wheel group:

usermod -a -G wheel $USER_NAME

And as always when you make changes, test your work! Login as the user:

ssh $USER_NAME@host.domain.tld

And try to escalate to root:

su -

If you were successful, you may now disable root logins in the sshd configuration file.

In /etc/ssh/sshd_config, change PermitRootLogin to no such as below:

PermitRootLogin no

Or if you REALLY need a copypastable method of making that change without errors:


(v_FIRST_MATCH="$( egrep -n -m1 "^[[:blank:]]*Match" /etc/ssh/sshd_config | cut -d ":" -f1 )"
if [[ -z $v_FIRST_MATCH ]]; then 
   v_FIRST_MATCH="$( cat /etc/ssh/sshd_config | wc -l )" 
fi
v_PERMIT_ROOT_LOGIN="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^[[:blank:]]*PermitRootLogin" | awk '{print $2}' | head -n1 )"
if [[ -z $v_PERMIT_ROOT_LOGIN ]]; then 
   sed -i "$v_FIRST_MATCH i PermitRootLogin no" /etc/ssh/sshd_config
elif [[ $v_PERMIT_ROOT_LOGIN != "no" ]]; then 
   sed -i "0,/^[[:blank:]]*Match[[:blank:]]/{s/^\([[:blank:]]*PermitRootLogin.*\)/#\1\nPermitRootLogin no/}" /etc/ssh/sshd_config
fi)

And since we're changing configurations of a service, restart sshd!

(if [[ -z "$( which systemctl )" ]]; then 
   /etc/init.d/sshd restart; 
else 
   systemctl restart sshd; 
fi)

Disable Passwords

You can disable all passwords across the server (force SSH keys) by ensuring that the /etc/ssh/sshd_config file has a few specific directives set. Before making any changes, however, be sure that you have generated and tested an ssh key.

Here are the directives that you will need to set:

<source lang='text'> PasswordAuthentication no ChallengeResponseAuthentication no PubkeyAuthentication yes </source>

Protocol

This is the second most common OpenSSL vulnerability that PCI scans find. The most common is old versions of SSH, which is covered by backporting.

Often, SSH is set to allow SSH Protocol 1 or 2. This is disabled by specifying in /etc/ssh/sshd_config:

<source lang='text'> Protocol 2 </source>

You can verify this by ssh'ing to the server and using the -v flag - possible Protocols are listed as "Remote protocol version".

Alternate IP

When changing the IP for SSH on a server, please ensure that the Auth tab for the subaccount is updated, so that other techs know which IP to use to connect to the machine. This will change the copy-pasteable login info with the new IP on the main billing page. If you want to restrict SSH to listen only on a certain IP, you can do so by adding this line in /etc/ssh/sshd_config:

ListenAddress ip.ad.dr.ess:22

A certain port can also be specified by replacing the '22'. SSH can even listen to different ports on different addresses:

ListenAddress ip.ad.dre.ss1:2552
ListenAddress ip.ad.dre.ss2:2442

As always, ensure that these ports are opened inbound in the firewall for the appropriate IPs or interfaces, and restart sshd when finished editing the configuration.

OpenSSH Client

Your system-wide SSH client config file is at /etc/ssh/ssh_config. You can also create a user specific file (can override anything in the system level file) at ~/.ssh/config.

Strict Host Key Checking

When you SSH to a new server, often you will end up with a prompt that looks like:

jack@jack:~$ ssh root@deering
The authenticity of host 'deering (67.225.255.50)' can't be established.
RSA key fingerprint is cb:45:20:ac:e8:17:0b:7a:4f:63:d8:6d:78:91:6a:94.
Are you sure you want to continue connecting (yes/no)? 

This prompt is very useful - not just annoying. It tells you if you have already SSH'd to that OpenSSH server on that IP. It will also blow up in your face when the OpenSSH server on that IP looks different.

Because of the above, it is HIGHLY recommended that you do not disable this Some people like to hide this. If you want to, you can add the following to your ssh client config to automatically add host keys:
StrictHostKeyChecking no

With just that, if the key is not known SSH will automatically add it without asking you first - but still notify you.

Host sections

You can set up specific configuration for any user by using the Host directive - simply start a section with the directive Host and any servers it should match, and then put in any parameters you would like. Once you do this, you can use a few directives to set specific parameters - username to use, port to connect on, private key to use, hostname to connect to instead of the one provided, etc. It is conventional to indent anything under that Host's section - and this would be set up on your workstation. None of these are required.

Host myvps
    User USER
    Port 22222
    IdentityFile /home/jack/.ssh/user.com.key
    HostName jhayhurst.com

If you do this, you may want to do a last section at the bottom where you specify a wildcard - for all hosts that have not yet been matched.

Write-Protected: Broken Pipe

Sometimes you're disconnected you say? Disconnected from a server? Use ServerAliveInterval to send a "hey I'm still here" packet every once in a while. You can do this by adding to your SSH config:

<source lang='text'>

 #Value below in seconds.
 ServerAliveInterval 30
 #Value below is the number of times ServerAliveInterval is done per session.
 ServerAliveCountmax 1200 

</source>

30 seconds time 1200 times gives you 600 minutes, or 10 hours - enough to get you through a 10 hour shift. Of course, if you are going to run something that is going to take a while, you should still run it in a screen.

Shared SSH sockets

Normally, every time you run SSH, it opens a separate SSH tunnel to the server for that connection. You have to enter the password every time. Etc. It's kinda annoying.

Instead, if you would so like, you can share your SSH connections to the server. Then, to open the first SSH connection - the connection is opened as a socket, and any further SSH connections to that same server use that same socket. Thus, you have to enter the password for that given user only once, and there's only one socket open between the servers.

To set this up, add the following lines to your SSH config in the proper Host section:

<source lang='xorg_conf'> ControlPath ~/.ssh/.master-%r@%h:%p ControlMaster auto ControlPersist 1 </source>

Then, that makes things like this happen:

[root@server] dbstats >> lsof -i :22
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
sshd    2876 root    3u  IPv6   5873      0t0  TCP *:ssh (LISTEN)
sshd    2876 root    4u  IPv4   5875      0t0  TCP *:ssh (LISTEN)
sshd    8427 root    3u  IPv4 660136      0t0  TCP Servername.com:ssh->USERNAME.com:61215 (ESTABLISHED)
[root@server] dbstats >> w
 18:50:47 up 14 days, 20:27,  4 users,  load average: 0.60, 0.56, 0.68
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/1    USERNAME:S.0    Wed23    0.00s  0.43s  0.43s /bin/bash

SSH Keys


You do not put your private SSH key on a customer's server. None of your personal keys should be on a customer server, ever. SSH keys are paired, private to public, and can be used with OpenSSH to authenticate by key instead of password.

It is highly recommended that you password protect your keys. Then, if someone gets the key, you're not QUITE as screwed - they need the password too. However, password protecting a customer's key is less effective - we need to record the passphrase right next to the key, doing so negates a lot of the additional security.

SSH Key for access to server


Until Cent5 goes EOL, we will be using only RSA 4096 bit keys Generate the key You do this on your workstation, not on their server. Go into your ~/.ssh/ folder, and run the following command:

ssh-keygen -t rsa -b 4096 -C "$(whoami)@$(hostname)-$(date -u +%Y-%m-%d-%H:%M:%S%z)"

In the below example, make sure you're adjusting the location and keyname to suit the situation.

This will prompt you and end up looking like:

bash> ssh-keygen -t rsa -b 4096 -C "$(whoami)@$(hostname)-$(date -u +%Y-%m-%d-%H:%M:%S%z)"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): host.server.tld.key
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in host.server.tld.key.
Your public key has been saved in host.server.tld.key.pub.
The key fingerprint is:
07:51:de:4a:de:ce:89:2:d7:6d:4a:d7:14:ca:5e:92 root@host.server.tld
bash>


Install the key Next, in another terminal, SSH into the server. Pick which user you want to SSH in as, and then copy the contents of ~/.ssh/host.server.com.key.pub into ~/.ssh/authorized_keys for that user. So, if we wanted to SSH in as the user "lwstaff" on the remote server, we would edit the file /home/lwstaff/.ssh/authorized_keys. The contents of that file will be one long line, and just put it on a new line in that file. If root, it's gonna go into /root/.ssh/authorized_keys.

That line is probably going to look like:

ssh-rsa  AAABB3NzaC1kc3MAAACBAJP+AGzwlQlnzu8Omj8aE53Gec0LlB8EaYlInqE0HFLRS7KXlWhVLyNBl8VsDDey0fvkatu1XqTU8phm6ZA9xxGzUcyah8nm9c9cF+aFb38a1B5+YgVXKiEAty8HzbZxHtmfDXR/irKUNv71rmxXQp6dvQp08+JsWQQjV65kUj17AAAAFQDWbK2P6Kw0zKgeePw/lgvVJRBrIQAAAH89mM9RMJtr4VjY5p+O+B/alGNYUOHMXVYRoMBFimSy6ibZ+MqHxFm2HLBlcGChe8WJqfgtZcIAGhqmKe24kzWqUywanoWiOqn4vQ+KXVLvnuD/WuyP2/zGv83CY4nuS9gSQcvSYMLxmjdj2CLpmYPCmG1Im/YFQOl2JEYgxLNtAAAAgHl0izFxatq1FE77qRUDXyj1fyfHRW1YU73cI9X4Sa4bmPwTbM2PmI7wL3FoKFxglWGwraspjcTMmriGMT0ZuZSJO2kvN7qI2w1fxZlEtOwQnT0JnC6A9U8F0JXzCsqYWMmF2dpmJuVc9Zg4yZalwCxVlCLP/qgQoZKA7Eyr6uro root@host.server.com

Alternately... and more easily:

ssh-copy-id -i ~/.ssh/host.server.tld.key USERNAME@IPORHOSTNAME

Test the key In another terminal, once you've saved that file, try SSH'ing into the server to verify your SSH key works. You should use something like this to test it:

ssh -i ~/.ssh/host.server.com.key root@host.server.com

Note the key Assuming the key worked, create a sticky note on the account. The format of this sticky note is:

Copy the following commands on your workstation terminal

cat <<EOM >~/.ssh/host.server.com.key
<put the contents of the key here>
EOM
chmod 600 ~/.ssh/host.server.com.key
ssh root@host.server.com -i ~/.ssh/host.server.key

So this ends up looking like:

Copy the following commands on your workstation terminal

cat <<EOM >~/.ssh/host.server.com.key
-----BEGIN RSA PRIVATE KEY-----
RIIEogIBAAKCAQEA3jC+Wt2ZUTxfmwXk8uUgg6yr5nOK5gKe1FOAvi/fKFwsbJcb
nuefdnV3YPXKdQJZeVe/uSfjQ8rb6+X3ZxqItV7Khyvh6ZZ6t/HGYY7aYd2zBcfa
OEt4LGuC/X39XfMGlXy0kOkDed0ybxbbN8/OaB3tmjQ/ZaWIwOqQVphYd/JsfLmR
j+0ns737yNwAzqOW36fT0rSNlGGsUk9NZB/jiz1wfy1YOIJYIyRcrYwV05wNYLku
3lc4cic2P6qc+FKVsUgP6flmd8jk4+GHbAI/7Iq90SgjSazbEQra5J+niScWbq4V
79DY1wVgNGnV+EGMpwr4flvYXx8/DsxmZGdnuQIBIwKCAQByRPQuuxumHwyY3nW+
vvrHYB3kO2wBQy0cvTrld7SYaer2BJHcsaJ3b5x7AhBZbuwvxrpfORXLF97RFy7H
T3mB2PpxZwZ4IYDw4r3Lxc9lh/WrR5TMfpWTL/o5OXr9HeYg/k475YzCVIBWY4au
/ShSzZAja1PN4Bpx17fyBTTP+A7SCelMw8FGeelUxnFA0H75kRo21GJfh3iUWwCY
5YROah1K2iZx9u1JUjepipNTJPR1AxGOKqLf/bKaVhaqnZDXkPOeaTzy+S9YfEyD
fMzgyeQdfpZcRCMObICfviArHJZppEWM4j7MyDw3AE0l/BMp84ivWV5ZIgR49PlX
b9njAoGBAP2WvGD2YQ0rky3ayUuUyx8XOffHujUrvRyResjn5AU4cgUvn+CPa2C8
jFN3gBtrvKUlspAeWdFL3ofWD9cvIq9QY1sBMcR58NH1yJUvPUIqdXSeoTN6dt7O
4fK3NI/bqwFuMo3AUMjU8coZbfJJuiyY/2MXqryE8pRLh3gBfhzrAoGBAOBNlJWg
uiK4BN2QV/V0tNY4fmH1srWScnqgXHUaC87WxDJac4I9RQ2Q6fJP8E23+xsenRkP
HEbkhgwEngt9b0y9vKJRM+ReSOGJJ0EENeEpcEaHZiDGeJumHQqFB6A9NkXdvB1k
q7JXccuwCYaPVh5Y6u8uLgDXhvdmTDYAAZTrAoGBAO8ZGAOmeMNGV5GiZgVvCKhJ
GWX21COBAsMqFK7MAuBZypcs5zLQXe1+ku+cjrrEqo0U6jAcndP3E6SlQiKoyO5w
Xa2TaXAbM4QiQMcsijcSFvjtVik487wiJX5yON9oty08A8d6z9NMbvHANHa6ivbZ
ZdJ1Znc7haHDjls0m3pZAoGAOa2Mm4EZ66upes1YclE10K9w9J5D5YwOzxNK+Ypa
zsmC6GBm2Fjl3uq/0Jg2emKCZg8vtfySPh2Bjg/Q3mIVTj9rBSrS1FLCSJ+cYSz3
6XEOPgWPSkGp/CNmja0t2L9IeF2WxbrNEJokk3Zo2XVQqLfHYhMwZp3SP532UuoP
CQkCgYEAwAQZAdaQuwUV39Q6kathxKR04DHkelJW+tBIMpmWXbGVyEOqvXbGckF+
WPmEL/ScoP+tNJN0Lyl6uI7zBXGeTQOfQUI1iXOcS52AeZ6AWVw/NCFGl0nbQNSD
h/iUEQjuRy5YtNYlK5HScRCrInX0C5qhHSMipdQrlZ6gMzVqppE=
-----END RSA PRIVATE KEY-----
EOM
chmod 600 ~/.ssh/host.server.com.key
ssh root@host.server.com -i ~/.ssh/host.server.key

This should be a sticky note on their account. Next, go into the auth settings for that subaccount, and under the special instructions section, add that SSH command in. So, you should be adding:

ssh -i ~/.ssh/host.server.key root@host.server.com

Generate a Key for billing

Some servers have ssh password disabled. Set up a keypair and put it in their Alternate Auth for the server.

A script for this to run on server:

curl -s files.sysres.liquidweb.com/sysres-toolbox/sysres-toolbox.sh > ~/sysres-toolbox.sh ; . ~/sysres-toolbox.sh
billing_ssh_key

Will provide all the needed output.

Or on your workstation:

curl -s files.sysres.liquidweb.com/sysres-toolbox/sysres-toolbox.sh > ~/sysres-toolbox.sh ; . ~/sysres-toolbox.sh
billing_ssh_key UNIQID

relevant function:

billing_ssh_key() {
	#ssh-key for billing
	#set uniq_id of server (used for key name)
	local uniq
	if [ $1 ]; then
		uniq="$1"
	elif [ -f /usr/local/lp/etc/lp-UID ]; then
		uniq=`cat /usr/local/lp/etc/lp-UID`
		local on_server=1
	else
		echo "input uniq id of server from billing (ex: A1B2C3)"
		read uniq
	fi
	echo "Using $1 for key name"

	# check for existing keyfiles:
	if [ -f ~/.ssh/$uniq ] || [ -f ~/.ssh/${uniq}.pub ]; then
		echo "detected previously created ssh keys at ~/.ssh/$uniq{,.pub} !"
		return 1 
	fi

	echo "generating passphrase for ssh-key..."
	pass=`tr -dc '_A-Z-a-z-0-9!@#$%^&*()[];:,.<>/?|~' < /dev/urandom | head -c16`
	echo "$pass"

	echo "creating ssh key ~/.ssh/$uniq ..."
	echo
	ssh-keygen -f ~/.ssh/$uniq -N "$pass" -C "liquidweb staff"
	echo
	if [ -f ~/.ssh/$uniq ] && [ -f ~/.ssh/${uniq}.pub ]; then
		echo "ssh key pair created successfully!"

		pub=`cat ~/.ssh/${uniq}.pub`
		priv=`cat ~/.ssh/$uniq`
		if [ $on_server ]; then
			echo "adding Public key to ~/.ssh/authorized_keys :"
			echo $pub >> ~/.ssh/authorized_keys
			echo "done."
		else 
			echo "public_key to add to server:"
			echo "$pub"
		fi
		echo "ssh-key pass: $pass"
		# echo "private key ( add to billing comments with pass ): "
		# echo "$priv"

		echo "
cat <<EOM > ~/.ssh/$uniq
$priv
EOM
chmod 600 ~/.ssh/$uniq
ssh root@`hostname` -i ~/.ssh/$uniq
Passphrase: $pass
"
	else
		echo "ssh keypair not detected... please attempt manually!"
	fi
}

Changing the port, disabling root logins, disabling passwords, and generating a key ALL AT ONCE

Template:Cleanup If you need to make numerous changes to how a customer's server is accessed all at once, there's an internal google doc that guides you through this process: (link commented out and contents pasted below.


How to: Create a new Linux user Generate an ssh key to login with that user Change the ssh port Disable ssh root logins Disable ssh password auth Enable ssh public key auth NOTE THE ACCOUNT Add the new port to apf or csf Restart the necessary services Test ssh logins


Assumptions: This document assumes that the server in question is running CentOS 5, 6, or 7, and is running cPanel. Mileage may vary on servers set up differently than this. This document also has instructions that are specific to the apf and csf firewalls - If you're using something other than these, you will need to investigate how to open ports in whatever it is that you're using. This document assumes that you are making *all* of the changes listed above, however there are instructions (highlighted in red) telling you what to skip in case you are not. IMPORTANT - The commands (highlighted in green) provided in the document assume that all previous commands have been run (In some instances there are alternate commands (highlighted in blue) where you can run one or the other, or commands that are specifically specified as being optional), and that the user has evaluated whether or not they completed successfully. The reason behind this is because some of the commands store variables that will be used by other commands later in the document. You don't have to run these commands - the instructions themselves should adequately describe how to make the necessary changes - the commands are just there to make it easier. If anything goes awry, there's a section at the end of the document on rolling back changes.


Things that we will need to do: Create a new linux system user - with root logins disabled, we will need to first login as another user, and then change to the root user in order to perform any tasks. Create a private / public key pair for that user - If we're not using password authentication, we will need to authenticate by another means - using an ssh key. Add the public key to the list of authorized keys - without this in place, having the key will do us no good. Note the account (THIS IS SUPER IMPORTANT). Modify the sub-account (THIS IS ALSO SUPER IMPORTANT). Make changes to the /etc/ssh/sshd_config file - This is where the details are kept regarding how ssh functions. Open the new port in the firewall - If we changed the ssh port, we need to make sure that traffic to that port can get through the firewall. Restart sshd - Or the changes won't take place. Restart the firewall - Or the changes won't take place. Test everything.


Things that the customer needs to be aware of (IMPORTANT): Disabling root logins will impact how they access their server - If they or any of their users login to ssh with the root user, this will impact their ability to access. Disabling password authentication will impact their users - If their users or customers access their server by ssh, or user other systems that utilize the ssh protocol (sftp, scp, rsync, sshfs, etc.), disallowing password access will complicate and potentially prevent their ability to access the server. While disabling both ssh root logins and password authentication is generally a best practice, for the most part, having csf and lfd present on a server makes it less likely that either of these will be an issue - as is evidenced by the fact that the vast majority of the root level compromises we see (which are fairly minimal to begin with) usually appear to be related to a compromise on the user's end rather than a brute force attempt. You should check to make sure whether csf or apf is installed:

if -z $( which -a csf ) && -z $( which -a apf ) ; then echo -e "\nNeither csf nor apf appear to be installed - recommend installing csf.\n"; elif -z $( which -a csf ) ; then echo -e "\napf appears to be installed - recommend upgrading to csf.\n"; elif -z $( which -a apf ) ; then echo -e "\ncsf appears to be installed.\n"; else echo -e "\nBoth csf and apf appear to be installed. This can cause issues. Please confirm with the customer and then remove apf.\n"; fi

Making modifications to how the server is accessed has the potential of impacting the speed of our support - The point of these changes are to make it more difficult for an unauthorized user to gain access, but they might also make it more difficult for US (or anyone who might be providing support for them) to gain access in the event of a critical situation. We have a system to help us ensure that this impact is minimal, but they need to be aware that the potential for this impact exists none-the-less.


Why not a script? Technical understanding - Running a script to get the results that you're looking for is all well and good, but it's better to have a technical understanding of the scope of the changes that are being made as well as the reason behind them. It is the goal of this document to help provide that understanding. Error checking - While it's difficult for a script to ensure that all commands it runs turn out as expected, it's easy for a human being to review the results and output from the commands and validate that the desired result has been achieved.


What if the customer just wants the warnings to go away? Actually, if the customer doesn't want to make any changes, but also does not want to receive further warnings from cPanel regarding the data in the security advisor, there's a simple script that can accomplish EXACTLY this effect:

wget -qO ./cp56_disable_sec_adv.sh http://layer3.liquidweb.com/acwilliams/cp56_disable_sec_adv.sh; chmod +x ./cp56_disable_sec_adv.sh; ./cp56_disable_sec_adv.sh


Creating a new linux user: Choose a username - It's bad form to use the same username for each server, so we should either be asking the customer for their preference or randomly generating a username. Here's a command to randomly generate one:

v_USERNAME="lw_$( cat /dev/urandom | tr -dc 'a-z' | fold -w 8 | head -n1 )"; echo -e "\n$v_USERNAME\n"

Or you can just declare whatever you want:

echo;read -ep "What username do you want to use? " v_USERNAME; echo -e "\n$v_USERNAME\n"

Verify that that username is okay to use - It's probably fine, but we need to make sure that the username we've selected doesn't already exist, and that there isn't a folder in the home directory with the same name.

if | $( cat /etc/passwd | grep -c "^$v_USERNAME:" ) -eq 1 ; then echo -e "\nCan't use \"$v_USERNAME\".\n"; else echo -e "\nGo ahead and use \"$v_USERNAME\".\n"; fi

Create the user using the "useradd" command. The "-d" flag is used to specify where their home directory is. Also, we'll want to specify that they're included in the wheel group (using "-G wheel") and that they use bash as their shell (using "-s /bin/bash"). In addition to this, we need to create a ".ssh" directory for them in their home directory and make sure that the permissions for that directory are "711". This command will accomplish that all of that:

useradd -d "/home/$v_USERNAME" -G wheel -s /bin/bash $v_USERNAME && mkdir -p "/home/$v_USERNAME"/.ssh/ && chmod 711 "/home/$v_USERNAME"/.ssh/ && chown $v_USERNAME:$v_USERNAME "/home/$v_USERNAME"/.ssh/


Create a key pair and authorize the public key: It's important at this point to know at least a little about how ssh keys work. They consist of a pair of files, one is called the "public key", the other is called the "private key". The private key, as is implied by the name, is secret - it's the equivalent of a password. Only people who can be trusted to access this server should know the contents of this file. You shouldn't store it anywhere where it might be seen by someone who isn't authorized. The public key, on the other hand, is stored on the server to check against the private key. If the server determines that the two of them match, the user is granted access. There are two optional choices that you can make here: Option 1) The private key can be encrypted with a password which you will be prompted for when attempting to access the server. Option 2) If you've decided not to disable password authentication, you won't need an ssh key to log into the server, just a password. In either case, you will need to either generate or decide upon a new password. The command below will generate a random 14 character password with no ambiguous characters:

v_PASSWORD="$( cat /dev/urandom | tr -dc 'a-km-zA-GJ-NPRT-Z346-9\-+_@#%^&~:;[]<>./?=' | fold -w 14 | head -n1 )"; echo -e "\n$v_PASSWORD\n"

Alternately, you can just specify your own password:

echo;read -ep "What ssh key password do you want to use? " v_PASSWORD; echo -e "\n$v_PASSWORD\n"

Run the command to create a key. The below command uses the "-b" flag to set the number of bits for the encryption (2048 or 4096 are considered standard). The "-f" flag tells the command where to create the file. The "-N" flag tells the command to encrypt the key with a password (If you didn't use the commands above to specify a password, the below command will leave the password blank), and the "-C" command allows the user to specify a comment (so that the public portion of the key will be human recognizable).

ssh-keygen -t rsa -b 4096 -f "/home/$v_USERNAME"/.ssh/ssh_key.key -N "$v_PASSWORD" -C "liquidweb-$(date -u +%Y-%m-%d-%H:%M:%S%z)"; v_KEY_CREATED=true

Alternately, if you've decided not to disable password authentication, you will need to assign the password that you created in the above step to this user:

echo "$v_PASSWORD" | passwd --stdin $v_USERNAME

The command "ssh-keygen" command above should have created two files, one named "ssh_key.key", which contains the private key, the other named "ssh_key.key.pub", which contains the public key. Authorize the public key by adding it to the user's authorized_keys file (.ssh/authorized_keys - It probably won't be there yet, so you'll need to create it). By putting the public key into this file, you're telling the ssh server (sshd) that if someone has the matching private key, they're allowed to access that user on the server. Make sure that that file has the correct permissions (600) and is owned by the new user as well. If you've decided not to disable password authentication, you will not need to take this step.


touch "/home/$v_USERNAME"/.ssh/authorized_keys; chmod 600 "/home/$v_USERNAME"/.ssh/authorized_keys; chown $v_USERNAME:$v_USERNAME "/home/$v_USERNAME"/.ssh/authorized_keys; cat "/home/$v_USERNAME"/.ssh/ssh_key.key.pub >> "/home/$v_USERNAME"/.ssh/authorized_keys


Find a port and IP address to connect to: If you're changing the port, you need to change it to something that isn't already in use. This command will give you a random number between 10000 and 39999:

v_NEW_SSH_PORT="$( cat /dev/urandom | tr -dc '0-9' | fold -w 5 | grep "^[1-3]" | head -n1 )"; echo -e "\n$v_NEW_SSH_PORT\n"

Alternatively, if you know what port you want to use already - for example, if you want to keep the port the same as it is now - just specify it like this.

echo;read -ep "What port do you want to connect to ssh on? " v_NEW_SSH_PORT; echo -e "\n$v_NEW_SSH_PORT\n"

Verify that the port you've selected isn't in use by something else. You can run "netstat -lpn" and check against the list of ports already in use to ensure that this is the case:

if awk '{print $4}' | egrep -o ":[0-9]{2,5}" | sort | uniq | sed "s/^://" | grep -c "^$v_NEW_SSH_PORT$" ) -eq 1 ; then echo -e "\nPort $v_NEW_SSH_PORT is already in use.\n"; else echo -e "\nPort $v_NEW_SSH_PORT is okay to use.\n"; fi

Regarding the /etc/ssh/sshd_config file - it's important to note that sshd can be fine tuned to specific situations using the "Match" directive. You can read up on this on the man page, but for this process, all you need to know is that if there are any lines that begin with "Match", we need to ignore everything in the file below the first of those lines, and that we shouldn't be editing anything in the file below those lines. This command will find what line the first "Match" directive is on if any are present, otherwise, it will find the last line of the file.

v_FIRST_MATCH="$( egrep -n -m1 "^blank:*Match" /etc/ssh/sshd_config | cut -d ":" -f1 )"; if -z $v_FIRST_MATCH ; then v_FIRST_MATCH="$( cat /etc/ssh/sshd_config | wc -l )"; echo -e "\nNo \"Match\" directives.\n"; else echo -e "\nThe first \"Match\" directive is on line $v_FIRST_MATCH\n";fi

Verify the IP address that we'll be connecting to - You're already connected to the server, so you probably know what IP address this is, but you should check /etc/ssh/sshd_config to verify that there isn't a "ListenAddress" directive set. If there is, we will need to connect to that IP specifically. Remember, don't bother looking further down in the file than the beginning of any "Match" directives.

v_IP_ADDRESS="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^blank:*ListenAddress" | awk '{print $2}' | grep "\." | head -n1 )"; if | $v_IP_ADDRESS == "0.0.0.0" ; then echo -e "\nNo IP Address specified in the file.\n"; else echo -e "\nThe IP address specified is \"$v_IP_ADDRESS\".\n"; fi

If the file doesn't have the "ListenAddress" directive at all, or if the directive specifies "0.0.0.0" as the IP, It means that sshd is configured to listen on all IPs that the server has. If you're unsure of what IPs the server has, you can either check the output of "netstat -nap" to see what address any existing connections might be coming in on, or check the output of the "ifconfig" command to see what IPs are present. Keep in mind that just because an address is specified in the file as the internal address for sshd to listen on, that does not necessarily mean that that's the correct IP address for a remote connection to be made to. If the server is natted behind a firewall, or otherwise has static routing applied to its network connections, it's possible that the connection will need to be made to an IP that isn't even listed as being on the server. Since you are already connected to the server, it's generally simplest to just state what IP address you used to make that connection:

echo;read -ep "What IP address do you want to connect to ssh on? " v_IP_ADDRESS;echo -e "\nIP Address: \"$v_IP_ADDRESS\".\n"

Alternately, you can check ifconfig as described above, and then verify that that IP is viewable externally. The command below should work the vast majority of the time.

v_IP_ADDRESS="$( ifconfig | egrep -o "inet( addr)*[ :][0-9.]*" | grep -v "127.0.0.1" | head -n1 | grep -o "[0-9.]*$" )";if -n $v_IP_ADDRESS ; then v_IP_ADDRESS="$( wget --timeout=5 --bind-address=$v_IP_ADDRESS -qO /dev/stdout 'https://api.ipify.org' 2> /dev/null )";fi; if -n $v_IP_ADDRESS ; then echo -e "\nIP Address: \"$v_IP_ADDRESS\".\n"; else echo -e "\nNo IP address found.\n"; fi

While we're looking at the sshd_config file, we should take note of what the settings are for the port it's listening on, root logins, password authentication, public key authentication, and challenge-response authentication. It's possible for any or all of these to come up blank. If that's the case, it just means that the configuration file doesn't have them specified, and so sshd is using the default settings ("22", "yes", "yes", "yes", and "yes", respectively).

v_SSH_PORT="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^blank:*Port" | awk '{print $2}' | head -n1 )"; echo -e "\nCurrent Port: \"$v_SSH_PORT\"\n"

v_PERMIT_ROOT_LOGIN="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^blank:*PermitRootLogin" | awk '{print $2}' | head -n1 )"; echo -e "\nRoot Logins: \"$v_PERMIT_ROOT_LOGIN\"\n"

v_PASSWORD_AUTHENTICATION="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^blank:*PasswordAuthentication" | awk '{print $2}' | head -n1 )"; echo -e "\nPassword Authentication: \"$v_PASSWORD_AUTHENTICATION\"\n"

v_KEY_AUTH="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^blank:*PubkeyAuthentication" | awk '{print $2}' | head -n1 )"; echo -e "\nPublic Key Authentication: \"$v_KEY_AUTH\"\n"

v_CR_AUTH="$( head -n $v_FIRST_MATCH /etc/ssh/sshd_config | egrep "^blank:*ChallengeResponseAuthentication" | awk '{print $2}' | head -n1 )"; echo -e "\nChallenge-Response Authentication: \"$v_CR_AUTH\"\n"


NOTE THE ACCOUNT (SUPER IMPORTANT): We need to make sure that the details for how to connect end up noted on the account. If they do not, we will almost certainly be unable to remotely access the server later. The note needs to include the private key that we generated earlier, the username, and the port that we'll be connecting to. The command below should output exactly what we need to note:

echo -e "\n# Copy / paste the following to log in to $(hostname)($v_IP_ADDRESS):\n"; if $v_KEY_CREATED == true ; then echo -e "cat <<EOM >~/.ssh/remotehost.key"; cat "/home/$v_USERNAME"/.ssh/ssh_key.key; echo -e "EOM\nfor i in \$( seq \$(tput lines) ); do echo -en \"\\\\033[1A\\\\033[K\"; done\nchmod 600 ~/.ssh/remotehost.key"; v_KEY_LOC="-i ~/.ssh/remotehost.key"; fi; if -n $v_PASSWORD && $v_KEY_CREATED == true ; then echo -n "echo -e \"\n\e[1;31mThe key's password is:\n$v_PASSWORD\e[00m\";"; elif -n $v_PASSWORD ; then echo -n "echo -e \"\n\e[1;31mThe password for $v_USERNAME is:\n$v_PASSWORD\e[00m\";"; fi; echo "echo -e \"\n\e[1;31mrun 'su -' and enter the root password for root access\e[00m\n\""; echo -e "ssh $v_USERNAME@$v_IP_ADDRESS -p$v_NEW_SSH_PORT $v_KEY_LOC\n"

Make sure that that note is stickied. If it can be avoided, the connection details SHOULD NOT go into the ticket, either as a note, or as a response to the customer. We rely on an external resource for our ticketing system - if we shouldn't be saving passwords there, then ssh keys shouldn't go there either. The best places to note this are on the customer's account, and in our internal pastebin. Unfortunately, if the customer needs to have that key, the ticket is the best way for us to provide it to them. In addition to this, we need to make sure that on the sub-account page, under "Auth Settings" > "Edit Auth Settings", we enter the new username, the alternate ssh port, the correct IP address, and leave special instructions stating that there's an account note containing the relevant RSA key.


if | $v_PASSWORD ; then echo -e "\nssh username: $v_USERNAME\nssh port: $v_NEW_SSH_PORT\nssh IP: $v_IP_ADDRESS"; if $v_KEY_CREATED == true ; then echo -e "special instructions: Use the ssh key in the sticky note from $( date +%F )\n"; else echo -e "password: $v_PASSWORD\n"; fi; else echo -e "\n\e[1;31mNo ssh key was generated, and also no password was created. SOMETHING IS WRONG HERE.\e[00m\n"; fi


Modify the sshd_config file: Before modifying the /etc/ssh/sshd_config file we need to make a backup of it.

v_TIMESTAMP="$( date +%s )"; cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.ssh_secure.$v_TIMESTAMP.bkp

Also before making any changes, it's best for us to run "sshd -t". This command reviews the sshd_config file and outputs details on any errors that it finds. If there are errors present, making changes to it might just compound those errors, and restarting sshd might result in the server being inaccessible, so if this command results in any output, you want to investigate and attempt to correct what it's reporting before going forward.

sshd -t

First, we need to change the port that sshd is listening on to our new port (if it's already listening on a port other than 22, we can probably leave it as is). This will, conveniently, be stored in a line that begins with "Port", if no such line exists, add it (above any "Match" directives) in the following format: "Port 1234". The commands below will only change the port if it's different than the one you decided on earlier.

if -z $v_SSH_PORT ; then sed -i "$v_FIRST_MATCH i Port $v_NEW_SSH_PORT" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); elif $v_SSH_PORT -ne $v_NEW_SSH_PORT ; then sed -i "0,/^blank:*Matchblank:/{0,/^blank:*Portblank:/{s/^\(blank:*Portblank:.*\)/#\1\nPort $v_NEW_SSH_PORT/}}" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); fi

Next, we need to disable root logins. This is declared with the "PermitRootLogin" directive. The config file likely already says "PermitRootLogin yes" - we can change this to "PermitRootLogin no". Once again, if the directive isn't present, you can add it above any "Match" directives. If you don't want to disable root logins, just don't follow this step.

if -z $v_PERMIT_ROOT_LOGIN ; then sed -i "$v_FIRST_MATCH i PermitRootLogin no" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); elif $v_PERMIT_ROOT_LOGIN != "no" ; then sed -i "0,/^blank:*Matchblank:/{s/^\(blank:*PermitRootLogin.*\)/#\1\nPermitRootLogin no/}" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); fi

We want to disable password authentication by changing the "PasswordAuthentication" directive. As above, if it says "yes", change it to "no". Once again, if you see this declared below the first "Match" directive, don't change it there - only above. If it isn't above any "Match" directives, add it. If you don't want to disable password authentication, just don't follow this step.

if -z $v_PASSWORD_AUTHENTICATION ; then sed -i "$v_FIRST_MATCH i PasswordAuthentication no" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); elif $v_PASSWORD_AUTHENTICATION != "no" ; then sed -i "0,/^blank:*Matchblank:/{s/^\(blank:*PasswordAuthentication.*\)/#\1\nPasswordAuthentication no/}" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); fi

If the "ChallengeResponseAuthentication" setting is set to "yes" (or not present and therefore at its default), we need to change it to "no". This is a feature that's not frequently used anyway, and without it disabled, cPanel will refuse to believe that password authentication is really turned off. If you don't want to disable password authentication, just don't follow this step.

if -z $v_CR_AUTH ; then sed -i "$v_FIRST_MATCH i ChallengeResponseAuthentication no" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); elif $v_CR_AUTH != "no" ; then sed -i "0,/^blank:*Matchblank:/{s/^\(blank:*ChallengeResponseAuthentication.*\)/#\1\nChallengeResponseAuthentication no/}" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 )); fi

While this probably isn't the case, it's possible that public key authentication has been disabled in the sshd_config file. This defaults to "yes", so if the "PubkeyAuthentication" directive isn't declared, there's no need to change anything. Only if it's set to "no" do we need to change it to "yes".

if -n $v_KEY_AUTH && $v_KEY_AUTH != "yes" ; then sed -i "0,/^blank:*Matchblank:/{s/^\(blank:*PubkeyAuthentication.*\)/#\1\nPubkeyAuthentication yes/}" /etc/ssh/sshd_config; v_FIRST_MATCH=$(( $v_FIRST_MATCH + 1 ));fi

Some of our internal systems require root access via public key, so we'll need to add a match condition to ensure that that access isn't blocked:

sed -i '/^blank:*Match user root address.*10.20.4/,+3d' /etc/ssh/sshd_config; echo -e "Match user root address 10.10.4.0/23,10.20.4.0/22,10.20.7.0/24,10.30.4.0/22,10.30.2.0/24,10.30.104.0/24,10.50.9.0/27\n PermitRootLogin yes\n PubkeyAuthentication yes\n AuthenticationMethods publickey" >> /etc/ssh/sshd_config

If you've just been copying and pasting these commands, double check the /etc/ssh/sshd_config file to verify that they have had the desired effect. It wouldn't hurt to use "sshd -t" to check it again as well.

sshd -t

Remember, we created a backup of the file. If you need to roll back, for any reason, there's a section at the end of this document that covers that.


Open the new port in the firewall: If you opted not to change the port, none of this section is necessary. Go ahead and skip to restarting services below. If csf is present, back up the /etc/csf/csf.conf file.

cp -a /etc/csf/csf.conf /etc/csf/csf.conf.ssh_secure.$v_TIMESTAMP.bkp

Alternatively, if the server is running apf, backup the /etc/apf/conf.apf file.

cp -a /etc/apf/conf.apf /etc/apf/conf.apf.ssh_secure.$v_TIMESTAMP.bkp

In csf, modify the "TCP_IN" line to remove the old port and include the new port in the list of comma separated ports:

v_TCP_IN="$( grep "^blank:*TCP_IN" /etc/csf/csf.conf | head -n1 )"; if [[$( echo $v_TCP_IN | egrep -c "[,\"']blank:*$v_SSH_PORTblank:*[,\"']" ) -eq 1 ]]; then v_NEW_TCP_IN="$( echo "$v_TCP_IN" | sed "s/\([,\"']blank:*\)$v_SSH_PORT\(blank:*[,\"']\)/\1$v_NEW_SSH_PORT\2/" )"; else v_NEW_TCP_IN="$( echo "$v_TCP_IN" | sed "s/,/,$v_NEW_SSH_PORT,/" )"; fi; sed -i "s/^$v_TCP_IN$/#$v_TCP_IN\n$v_NEW_TCP_IN/" /etc/csf/csf.conf

Alternately, if it's apf, modify the "IG_TCP_CPORTS" line to remove the old port and include the new port in the list of comma separated ports:

v_TCP_IN="$( grep "^blank:*IG_TCP_CPORTS" /etc/apf/conf.apf | head -n1 )"; if [[$( echo $v_TCP_IN | egrep -c "[,\"']blank:*$v_SSH_PORTblank:*[,\"']" ) -eq 1 ]]; then v_NEW_TCP_IN="$( echo "$v_TCP_IN" | sed "s/\([,\"']blank:*\)$v_SSH_PORT\(blank:*[,\"']\)/\1$v_NEW_SSH_PORT\2/" )"; else v_NEW_TCP_IN="$( echo "$v_TCP_IN" | sed "s/,/,$v_NEW_SSH_PORT,/" )"; fi; sed -i "s/^$v_TCP_IN$/#$v_TCP_IN\n$v_NEW_TCP_IN/" /etc/apf/conf.apf

Verify that the line within the /etc/csf/csf.conf or /etc/apf/conf.apf file looks correct. If the server is using a firewall other than csf or apf, this document doesn't cover the steps necessary to open the ports there.


Restart the firewall and sshd: IMPORTANT - After restarting sshd, you MUST not logout of the server until you have tested and verified that you are able to login with the instructions that you've noted on the account. Restarting sshd will not disrupt your current connection, which means that you will still be able to roll back changes if necessary, but once you logout, you will no longer be able to log in with the root user and password. The commands for this will be different depending on whether or not you're on Cent7. On Cent6 and below, services were restarted with the "service [name] restart" command, or alternatively "/etc/init.d/[name] restart". On Cent7, services are restarted with "systemctl restart [name]".

csf -r; apf -r; if -z "$( which systemctl )" ; then /etc/init.d/sshd restart; else systemctl restart sshd; fi


Verify that everything is working: Now that sshd has been restarted, you can test by making an ssh connection to localhost to ensure that logins to the new user are working as expected. You will need to remember to connect with the user you created, specify the port with the "-p" flag, and specify the ssh key that you generated with the "-i" flag. If the ssh key you generated earlier used a password, you will be prompted for that password.

if -n $v_PASSWORD ; then echo -e "\nHere's the key's password: $v_PASSWORD\n"; fi; ssh -o UseRoaming=no -o VerifyHostKeyDNS=no -o StrictHostKeyChecking=no $v_USERNAME@127.0.0.1 -p$v_NEW_SSH_PORT -i /home/$v_USERNAME/.ssh/ssh_key.key "echo" > /dev/null 2>&1; if $? -ne 0 ; then echo -e "\nConnection failed\n"; else echo -e "\nSuccess\n"; fi

IMPORTANT - Before you log out of the server (you won't be able to fix any issues once you do) the final test is to verify that you're able to login by copying and pasting the credentials that you noted in the account into a local terminal on your workstation. If this completed successfully, good job, you've done it! The last thing we need to do is remove the keys that we generated - it's more secure not to leave them laying around on the server.

rm -fv "/home/$v_USERNAME"/.ssh/ssh_key.key*; unset v_PASSWORD


Rolling back changes: DON'T DO THIS unless something is broken, or you otherwise want to roll back the work that you've already done. If you need to undo the changes that you made, it's thankful that we made backups! First, we'll want to revert the changes to the configuration files that we modified:

mv -f /etc/ssh/sshd_config.ssh_secure.$v_TIMESTAMP.bkp /etc/ssh/sshd_config; mv -f /etc/apf/conf.apf.ssh_secure.$v_TIMESTAMP.bkp /etc/apf/conf.apf; mv -f /etc/csf/csf.conf.ssh_secure.$v_TIMESTAMP.bkp /etc/csf/csf.conf

Next, we want to restart the firewall and sshd again, so that they can be working off of the old configurations:

csf -r; apf -r; if [[ -z "$( which systemctl 2> /dev/null )" ]]; then /etc/init.d/sshd restart; else systemctl restart sshd; fi

The last step on the server is to remove the user that we created using the "userdel" command. Because we've just created the user and there's no concern about the files within their home directory, it's recommended to use the "--remove" flag with this so that their home directory will be removed as well.

userdel --remove $v_USERNAME

Finally, we need to remove the note that we made to the account, and ensure that the special settings that we put in place on the sub-account page are returned to their previous values. IMPORTANT - Once all of this is done, before you log out, test to ensure that you are able to access ssh the same way that you were initially.

The document has a lot of details regarding the the specifics of the changes that need to be made in order to create a new ssh user, generate a key for them, note the details of that key on the customer's account, make modifications to the sshd_config file, and then test the changes.

Also noteworthy - it's written in such a way that you can literally just paste the sections of green text into your terminal and achieve the desired output.

Hashed Keys

If your ~/.ssh/known_hosts file outputs the hostname in a hashed format like so;

88	|1|rNMjbSOiccDtv2c8qfbDENzLMKh=|nU-aqGDAXASMnKreS4gBPC8mBiY= ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAvcXoPQ3SSxG9XjWdcQsRNSRBu9R0fWaQapcesMrja1t0Z9mXyU6igqZHxu3adi3OQSxDLb/UsIYiSZScoQnE0SyGy1XpzNuCMvQ9WUVv+6BkVx1VYTbhWDU+t/pT9D3tOuORoqo5HdcKub2ppmjWNqiifI3UVUFMWAfAVbqFG+PYphBcxHA75ZYu+Fwhn5Gsoz+grCpmnxf0cI3O1MDGEY7TeK/pertBPlKirX4lXxUdQ8fp/TTXl+GyHk21BQaRCm1SHE0MqyyJieQpcPRTliIFg5PLFcZtS9I7g7RWxv5l5tuhna+c9jzoZZkYmoxrlX63sa2XDwUHGWxP10yQa4w=

You've got the 'HashKnownHosts' entry set to "yes" in your sshd_config, so the hostnames aren't available in plaintext.

Troubleshooting

Down ssh with openssl errors

Template:Warning I have come across a few servers recently that have an issue with openssl that causes ssh to go down.

The issue can be found by consoling the server from the parent. Once inside the server an rpm -qa can be ran to find the installed openssl version.

  rpm -qa | grep openssl

If you see the below results, then the server can be fixed with the following instructions.

  openssl10-libs-1.0.1e-1.ius.centos6.x86_64
  openssl10-devel-1.0.1e-1.ius.centos6.x86_64
  openssl098e-0.9.8e-17.el6.centos.2.x86_64
  openssl10-1.0.1e-1.ius.centos6.x86_64

Edit the yum.conf and remove openssl* from the exclude line, dont forget to readd this once you have finished this process.

  vim /etc/yum.conf

After that is done you can run the following command to replace openssl10 with the correct openssl version.

  yum replace openssl10 --replace-with openssl

It will most likely complain about dependencies. You can hit yes on this. Then confirm the change and it will install the correct openssl.

You should see this information from the replace

================================================================================
 Package          Arch    Version                Repository                Size
================================================================================
Installing:
 openssl          i686    1.0.1e-16.el6_5.1      system-updates-released  1.5 M
 openssl-devel    x86_64  1.0.1e-16.el6_5.1      system-updates-released  1.2 M
Removing:
 openssl10        x86_64  1.0.1e-1.ius.centos6   @ius                     1.5 M
 openssl10-devel  x86_64  1.0.1e-1.ius.centos6   @ius                     2.1 M
 openssl10-libs   x86_64  1.0.1e-1.ius.centos6   @ius                     2.2 M
Installing for dependencies:
 keyutils-libs    i686    1.4-4.el6              system-base               20 k
 krb5-libs        i686    1.10.3-10.el6_4.6      system-base              769 k
 libcom_err       i686    1.41.12-18.el6         system-base               37 k
 libselinux       i686    2.0.94-5.3.el6_4.1     system-base              108 k
 openssl          x86_64  1.0.1e-16.el6_5.1      system-updates-released  1.5 M
 zlib             i686    1.2.3-29.el6           system-base               73 k

After that is done, restart sshd and it should start up normally.

Permission denied (publickey).

This issue existed on my server when setting up a Private Key through WHM and disabling Password Authorization then trying to SSH in via "ssh -i ~/.ssh/host.domain.com.key root@domain.com". For some reason /root was showing this.

root@host [~/.ssh]# ls -ld /root
drwxrwxr-x. 29 1000 1000 4096 Jul 22 02:28 /root/

Not sure how that happened this fixed the Permissions denied issue.

chmod 700 /root
chown root. /root

Another possible cause is the StrictModes setting in /etc/ssh/sshd_config (man page excerpt below):

StrictModes
Specifies whether sshd(8) should check file modes and ownership of the user's files and home directory before accepting login. This is normally desirable because novices sometimes accidentally leave their directory or files world-writable. The default is ''yes''.

From https://help.ubuntu.com/community/SSH/OpenSSH/Keys:

"Chances are, your /home/<user> or ~/.ssh/authorized_keys permissions are too open by OpenSSH standards. You can get rid of this problem by issuing the following commands:"

chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

ssh_exchange_identification: read: Connection reset by peer

This usually comes down to the source server blocking the connection in some way. Important points to check include CSF, /etc/hosts.allow, and /etc/security/access.conf. Some of these can be checked from WHM, including CSX, CSF, and the Hosts Access Control modules.

ssh on an unknown alternate port

If sshd has been moved off 22 but it was not documented properly, and we have access to whm, log in to whm. You can try restarting the SSH service from WHM, and it will display the last few lines of the log file. this should tell you the listening port.

If this fails, you can restart ssh in safe mode. append your url after "cpsess3019753617" with:

/scripts2/doautofixer?autofix=safesshrestart

This will put ssh on port 22. sshd will revert back to the configured port after restarting the service.