Recently, I discovered that SSH of my VPS server is constantly battered as follows.
Apr 06 11:15:14 abastro-personal-arm sshd[102702]: Unable to negotiate with 218.92.0.201 port 53768: no matching key exchange method found. Their offer: diffie>
Apr 06 11:30:29 abastro-personal-arm sshd[102786]: Unable to negotiate with 218.92.0.207 port 18464: no matching key exchange method found. Their offer: diffie>
Apr 06 11:45:36 abastro-personal-arm sshd[102881]: Unable to negotiate with 218.92.0.209 port 59634: no matching key exchange method found. Their offer: diffie>
Apr 06 12:01:02 abastro-personal-arm sshd[103019]: Unable to negotiate with 218.92.0.203 port 16976: no matching key exchange method found. Their offer: diffie>
Apr 06 12:05:49 abastro-personal-arm sshd[103066]: Unable to negotiate with 218.92.0.212 port 49130: no matching key exchange method found. Their offer: diffie>
Apr 06 12:07:09 abastro-personal-arm sshd[103077]: Connection closed by 162.142.125.122 port 56110 [preauth]
Apr 06 12:12:18 abastro-personal-arm sshd[103154]: Connection closed by 45.79.181.223 port 22064 [preauth]
Apr 06 12:12:19 abastro-personal-arm sshd[103156]: Connection closed by 45.79.181.223 port 22078 [preauth]
Apr 06 12:12:20 abastro-personal-arm sshd[103158]: Connection closed by 45.79.181.223 port 22112 [preauth]
Apr 06 12:21:26 abastro-personal-arm sshd[103253]: Connection closed by 118.25.174.89 port 36334 [preauth]
Apr 06 12:23:39 abastro-personal-arm sshd[103282]: Unable to negotiate with 218.92.0.252 port 59622: no matching key exchange method found. Their offer: diffie>
Apr 06 12:26:38 abastro-personal-arm sshd[103312]: Connection closed by 92.118.39.73 port 44400
Apr 06 12:32:22 abastro-personal-arm sshd[103373]: Unable to negotiate with 218.92.0.203 port 57092: no matching key exchange method found. Their offer: diffie>
Apr 06 12:49:48 abastro-personal-arm sshd[103556]: error: maximum authentication attempts exceeded for root from 98.22.89.155 port 53675 ssh2 [preauth]
Apr 06 12:49:48 abastro-personal-arm sshd[103556]: Disconnecting authenticating user root 98.22.89.155 port 53675: Too many authentication failures [preauth]
Apr 06 12:49:51 abastro-personal-arm sshd[103558]: error: maximum authentication attempts exceeded for root from 98.22.89.155 port 53775 ssh2 [preauth]
Apr 06 12:49:51 abastro-personal-arm sshd[103558]: Disconnecting authenticating user root 98.22.89.155 port 53775: Too many authentication failures [preauth]
Apr 06 12:49:53 abastro-personal-arm sshd[103561]: error: maximum authentication attempts exceeded for root from 98.22.89.155 port 53829 ssh2 [preauth]
Apr 06 12:49:53 abastro-personal-arm sshd[103561]: Disconnecting authenticating user root 98.22.89.155 port 53829: Too many authentication failures [preauth]
Apr 06 12:49:54 abastro-personal-arm sshd[103563]: Connection closed by 98.22.89.155 port 53862 [preauth]
Apr 06 12:50:41 abastro-personal-arm sshd[103576]: Invalid user from 75.12.134.50 port 36312
Apr 06 12:54:26 abastro-personal-arm sshd[103621]: Connection closed by 165.140.237.71 port 54236
Apr 06 13:01:26 abastro-personal-arm sshd[103702]: Connection closed by 193.32.162.132 port 33380
Apr 06 13:03:40 abastro-personal-arm sshd[103724]: Unable to negotiate with 218.92.0.204 port 60446: no matching key exchange method found. Their offer: diffie>
Apr 06 13:11:49 abastro-personal-arm sshd[103815]: Received disconnect from 165.140.237.71 port 50952:11: [preauth]
Apr 06 13:11:49 abastro-personal-arm sshd[103815]: Disconnected from authenticating user root 165.140.237.71 port 50952 [preauth]
Apr 06 13:19:08 abastro-personal-arm sshd[103897]: Unable to negotiate with 218.92.0.208 port 59274: no matching key exchange method found. Their offer: diffie>
Apr 06 13:33:36 abastro-personal-arm sshd[104066]: Received disconnect from 165.140.237.71 port 50738:11: [preauth]
Apr 06 13:33:36 abastro-personal-arm sshd[104066]: Disconnected from authenticating user ubuntu 165.140.237.71 port 50738 [preauth]
Apr 06 13:34:50 abastro-personal-arm sshd[104079]: Unable to negotiate with 218.92.0.204 port 44816: no matching key exchange method found. Their offer: diffie>
Apr 06 13:50:32 abastro-personal-arm sshd[104249]: Unable to negotiate with 218.92.0.206 port 27286: no matching key exchange method found. Their offer: diffie>
Apr 06 13:51:58 abastro-personal-arm sshd[104261]: Received disconnect from 165.140.237.71 port 50528:11: [preauth]
Apr 06 13:51:58 abastro-personal-arm sshd[104261]: Disconnected from authenticating user root 165.140.237.71 port 50528 [preauth]
Apr 06 14:01:25 abastro-personal-arm sshd[104351]: Invalid user from 65.49.1.29 port 18519
Apr 06 14:01:28 abastro-personal-arm sshd[104351]: Connection closed by invalid user 65.49.1.29 port 18519 [preauth]
As you can see, it is happening quite frequently, and I am worried one might break in at some point. Since SSH access guards users with root-access, it can be quite serious once penetrated. How do I harden against these kind of attacks? Because this is VPS, disabling SSH is a no-go (SSH is my only entry of access). Are there ways to stop some of these attackers?
As always, thanks in advance!
We can’t ever stop this kind of stuff, but with something like fail2ban you can set it up to block on too many failures.
Really though - ensuring your system is kept up to date and uses strong passwords or use a SSH keys is the best defence. Blocking doesn’t prevent them from trying a few times. Moving SSH to a non standard port will stop most of the automated attacks but it won’t stop someone who is dedicated.
Move SSH to non-standard port, make endlessh use the default port. Only use SSH keys. Only allow correct users (so eg. your user and git/forgejo). Use fail2ban to aggressively ban (redirect to default port, so 22) and report to abuseipdb everything that fails to authenticate first try (wrong user, password instead of key), has non-compatible ciphers (generally, only allow TLS1.3 etc.), or fails in any other way. Just be sure that if you accidentally get banned yourself (eg. Ctrl+C-ing during authentication), you can use another IP (eg. force v4) for connecting.
Nice list of suggestions, but implementing all of them feels a little over-the-top.
Tbh, I myself still have SSH on port 22. Firstly, because I’m lazy, and secondly … yeah that’s it. I’m honestly just lazy. But spam bots trying office/cookie123 are not a real threat, and anyone trying to actually target me will either have somehow acquired my key + password, use one of the probably many security issues that exist in the dozen services I selfhost, social engineer me into doing something (not saying I’ve given out my (old) KeePass password once, but it could be, as love makes blind (I still love her)), or just smash my kneecaps until I give out everything.
Thanks, I will try fail2ban. I am using ED25519 for ssh keys, it seems like it’s the best defense on the ssh side. Do you happen to know why this kind of attack is so prevalent?
I’m not them, but among other reasons they are looking to build botnets (cryptomining, dosing, mass crawling), and they are searching for hosts with low security (or if you just made a mistake)
- Disable passwordless login.
- Disable password login.
- Require SSH keys
- Move SSH port to non-standard port
- Reject connections to port 22
- Install and enable fail2ban
About the best you can do.
Don’t reject connections to port 22, honeypot it and ban on connection attempt.
honeypot
That’s a lot more work.
Using a nonstandard port doesn’t get you much, especially popular nonstandard ports like 2222.
I used that port once and just as much junk traffic and ultimately regretted bothering.
My experience running several ssh servers on uncommon nonstandard ports for over 10 years has been that it has eliminated all ssh brute forcing. I don’t even bother with fail2ban. I probably should though, just in case.
Also, PSA: if you use fail2ban, don’t try tab completing rsync commands without using
controlmaster
or you will lock yourself out.My two cents: Using a nonstandard ssh port is good for dumping bots. True, you can easily do a port scan against a server and easily find all open ports nbd. But most off-the-shelf bots are looking for standard ports to penetrate. I know that when I format and reinstall the test server, as soon as I change the ssh port, bot noise goes down significantly. So, for a simple config edit and about 2 minutes of time, it seems worth the effort. It’s just one layer tho. And yes, it goes without saying to pick a port other than 22, 222, 2222, etc.
How about 22222?
Oh, that one’s fine. Everyone knows that 5-digit ports add extra security, which is why WireGuard runs on port 51820 by default. You can verify this by checking the max port number, which is also 5-digits, computers just aren’t powerful enough to crack 6-digit port numbers, so Linux hasn’t bothered increasing it. /s
It gets rid of most of the login attempts for me. I don’t use a popular port though. Pick a 5 digit port so they have to put in some effort to find it.
Using a nonstandard port doesn’t get you much
Uhh… It gets you a lot. Specifically, unless you know the port you can’t connect… So hey, there’s that…
This community really says shit sometimes that makes me go cross-eyed…
The top-rated answer to this question on the Security StackExhange is “not really”. https://security.stackexchange.com/questions/189726/does-it-improve-security-to-use-obscure-port-numbers
On Serverfault, the top answer is that random SSH ports provide “no serious defense” https://serverfault.com/questions/316516/does-changing-default-port-number-actually-increase-security
Or the answer here, highlighting that scanners check a whole range ports and all the pitfalls of changing the port. Concluding: “Often times it is simply easier to just configure your firewall to only allow access to 22 from specific hosts, as opposed to the whole Internet.” https://security.stackexchange.com/questions/32308/should-i-change-the-default-ssh-port-on-linux-servers
And I’m a CEHv7. A literal security professional–and I say that an overwhelming vast majority of attacks against servers using SSH are going to come over the default port. Quite literally 99%. This means that you can lower your attack surface by exactly 99% by simply changing the default SSH port…
Those posts provide no meaningful insight and what they say is by the very technical of all interpretations is correct, I absolutely disagree with these statements. What they mean to say is that simply changing the default SSH port isn’t alone I means of strictly protecting yourself. Meaning you shouldn’t change the default SSH port and think that your server is secured because it’s not.
Quite the different interpretation than me saying it should be mandatorily a part of your security strategy.
In protecting yourself against port scanning is trivial.
Anyone underestimating the power of changing The default SSH port is someone who’s opinion I can safely disregard.
Do you have a source to cite for the literal 99%?
Reasoning skills and experience. There are entire botnets dedicated to finding servers with open SSH ports on 22. If the bots can connect, the IP of the server will be added to a list to be brute forced.
I’m a per diem linux systems administrator. Right now I have a VPS that I setup myself. It uses a non-standard ssh port, fail2ban, and rejects incoming connections to port 22. According to connection logs, I get about 200 attempts per 24 hours from bots randomly pinging ports to see if they can catch an open SSH port–and they’re banned via fail2ban.
I checked out some other servers that I manage, which I did not setup and have no control over how they operate. Sifting through just 3 random servers and checking connection logs, they have a combined 435,000 connection attempts in the past 6 hours between the 3 of them. These are relatively small servers with an extremely small presence. Simple fact of the matter is, is that they all have port 22 open and reachable. So botnets attempt to brute force them.
So just anecdotally that’s a difference of 0.0459770115% or 99.96%. Anyone telling you that changing the default SSH port doesn’t do anything for security has absolutely no practical experience at all. It significantly reduces your attack surface as bots have to guess at ports until they find your SSHd’s operational port to even begin to start sending attempts.
Move the ssh port to higher ranges, 30-60000. That alone will stop 99% of the attacks
Disable root logins, now usernames must be guessed too which will make success even lower
Then require SSH keys
At that point it’s like being in a nuclear fallout nshelter behind a 3 meter thick steel door and you can hear some zombies scratching on the outside… I’m not worried about any of that shit
For added funs run an SSH tarpit to fuck with the attackers, something like endlessh.
Well yeah, sure, but that doesn’t really add to your security and it only costs you work and resources
100% agree, that is a “totally for fun” exercise
This is what I do. Changing the port to a higher number will prevent almost all bots.
I understand that obscurity is not security but not getting probed is nice.
Also ssh keys are a must.
I do log in as root though.
However, I block all IPs other than mine from connecting to this port in my host’s firewall. I only need to log in from home, or my office, and in a crisis I can just log in to OVH and add whitelist my IP.
I do log in as root though.
Don’t do that. You’re one local piece of malware away from getting your server pwned. Logging in as an unprivileged user at least requires another exploit on the server to get root permissions.
For security disable password authentication - use public key instead, disable root login via ssh - use sudo or su from another user.
To reduce the number of attempts of others trying to get in change the ssh port and/or set-up fail2ban.
You could also set a firewall rule to only allow ssh from your IP address, if you have a static address at home and only need access from there, or have a way to VPN into your home network. Make sure you have a static address if you do this though, you don’t want your IP to change and be left locked out of your server.
You could also set a firewall rule to only allow ssh from your IP address
You can also broaden this to a region. You may still want to access SSH from various places around your country (e.g. when visiting family or friends), but likely won’t ever need to from most of the rest of the world, so block everything except IPs from your region (or regions you care about, e.g. any VPSs you have).
In addition to other advice you could also use SSH over Wireguard. Wireguard basically makes the open port invisible. If you don’t provide the proper key upfront you get no response. To an attacker the port might as well be closed.
Here’s at least one article on the subject: https://rair.dev/wireguard-ssh/
I generally do a few things to protect SSH:
- Disable password login and use keys only
- Install and configure Fail2Ban
- Disable root login via ssh altogether. Just change “permit root login” from “no password” to just “no”. You can still become root via sudo or su after you’re connected, but that would trigger an additional password request. I always connect as a normal user and then use sudo if/when I need it. I don’t include NOPASSWD in my sudoers to make certain sudo prompts for a password. Doesn’t do any good to force normal user login if sudo doesn’t require a password.
- If connecting via the same network or IPs, restrict the SSH open port to only the IPs you trust.
- I don’t have SSH internet visible. I have my own Wireguard server running on a separate raspberry pi and use that to access SSH when I’m away, but SSH itself is not open to the internet or forwarded in the router.
I vote for wireguard here, I don’t expose anything other than game servers to the internet
Disable passwords and use public private keys and don’t worry about it
This is the only answer you need to read. It’s a non-problem if you just do this, and there’s no reason not to do it.
Agreed!
OP, here is what I do. It might seem overboard, and my way doesn’t make it the best, or the most right, but it seems to work for me:
- Fail2ban
- UFW
- Reverse Proxy
- IPtraf (monitor)
- Lynis (Audit)
- OpenVas (Audit)
- Nessus (Audit)
- Non standard SSH port
- CrowdSec + Appsec
- No root logins
- SSH keys
- Tailscale
- RKHunter
The auditing packages, like Lynis, will scour your server, and make suggestions as to how to further harden your server. Crowdsec is very handy in that it covers a lot of ‘stuff’. It’s not the only WAF around. There is Wazuh, Bunkerweb, etc. Lots of other great comments here with great suggestions. I tend to go overboard on security because I do not like mopping up the mess after a breach.
ETA: just looked up one of your attackers:
218.92.0.201 was found in our database! This IP was reported 64,044 times. Confidence of Abuse is 100%: ISP CHINANET jiangsu province network Usage Type Fixed Line ISP ASN AS4134 Domain Name chinatelecom.cn Country China City Shanghai, Shanghai
busy little cunts.
No Port-knocking? Amateurs! /s
It’s absolutely overboard, and you can get 99% of the way there with this:
- WireGuard config (Tailscale in your case)
- Bind SSH to WireGuard IP only (so no public SSH port)
- SSH keys only, and disable root login over SSH
That will require breaking WireGuard and openSSH’s key-based authentication, which just isn’t happening. The rest looks like mostly auditing. Even a firewall isn’t necessary if no ports are accessible anyway (i.e. everything only accessible over Tailscale), and you can just configure iptables to block everything on the WAN IP and call it a day.
sugar_in_your_tea @sh.itjust.works
It’s nice to be commented by someone famous.
Open up the window, let some air into this room I think I’m almost chokin’ from the smell of stale perfume And that cigarette you’re smokin’ 'bout scare me half to death Open up the window, sucker, let me catch my breath
Mama told me not to come.
Fun fact, my usernames on Reddit (I would cycle them every couple of years) were all Three Dog Night lyrics, so I continued the theme on Lemmy.
Thanks for noticing. 😀
- harden sshd
- use fail2ban or even better
CrowdStrikeCrowdSec - use a tool like the following to have a next-gen security solution: https://github.com/mrash/fwknop
did you mean crowdsec instead of crowdstrike?
Fml… yes, I meant CrowdSec. Thanks for the hint
harden sshd
More details:
- require keys to login
- don’t allow login as root
That should be plenty, but you could go a bit further and restrict the types of algorithms allowed (e.g. disallow RSA if you’re worried about quantum attacks). For this, I recommend a subtractive config (e.g.
HostbasedAcceptedAlgorithms=-rsa-*
). This is way over the top since an attacker is unlikely to attack the cipher directly, but it could be part of an attack.Your answer to “how to harden SSH?” is “harden SSH”?
I know your two other points gave concrete suggestions, but it’s pretty funny you suggested to “harden sshd” when that is what OP is asking how to do.
Yeah, I see your point. No use to repeat the same you can read in other comments or in those 274772 guides online. I was trying to imply to just generally harden ssh because then brute-force attempts should be no issue, unless you log everything and the disk space gets maxed out :D
Configure the firewall with a IP whitelist to only allow connections to ssh be made from your home IP.
Other then that, disable password logon for ssh and setup up key based authentication.
Agreed, but be careful about the whitelist. If your home IP changes, you’ll be locked out until you update it, so you should consider an IP range if that’s a possibility for you. Likewise, if you’ll be accessing it from multiple locations (say, a family member’s house), then make sure to add those as well.
The best way is to disable password login and use SSH keys only. Any further steps are not required, but you may additionally install fail2ban or sshguard.
There’s a dedicated tool named sshguard which works nicely.
In addition to what others say, I also have ntfy notifications on successful login.
That’s genius. I’ll do the same from now on.
There is actually an example on their website.
What VPS are you using?
You should be able to setup a firewall, blocking all access to the SSH port. Then setup a VPN so that only you can access via SSH after making your VPN connection.
If you connect via a static IP, you can also create an ACL for the VPN connection just in case. You can set an ACL for the SSH port forward rule directly as well, but I don’t like that personally. I prefer keeping things behind the VPN.
Exactly, this I what alI do!
This is the correct answer. Never expose your SSH port on the public web, always use a VPN. Tailscale, Netmaker or Netbird make it piss easy to connect to your VPS securely, and because they all use NAT traversal you don’t have to open any ports in your firewall.
Combine this with configuring UFW on the server (in addition to the firewall from the VPS provider - layered defence is king) and Fail2Ban. SSH keys are also a good idea. And of course disable root SSH just in case.
With a multi-layered defence like this you will be functionally impervious to brute force attacks. And while each layer of protection may have an undiscovered exploit, it will be unlikely that there are exploits to bypass every layer simultaneously (Note for the pendants; I said “unlikely”, not “impossible”. No defence is perfect).
This is not “the correct answer”. There’s absolutely nothing wrong with “exposing” SSH.
You seem like a fan of the “pull out” method.
Public ssh is completely fine as long as you use key based auth only and keep your sshd up to date. Stop spreading bullshit.
A lot of things are “fine”, but the cost of adding Netbird to your VPS is effectively zero, whether counted in dollars, time, or convenience.
Given the massive security benefits of using a VPN, and the lack of any meaningful downside to doing so, you’d be an idiot not to.
This is moving the goal posts. You went from “ssh is not fine to expose” to “VPN’s add security”. While the second is true, it’s not what was being argued.
Never expose your SSH port on the public web,
Linux was designed as a multi user system. My college, Cal State Northridge, has an ssh server you can connect to, and put your site up. Many colleges continue to have a similar setup, and by putting stuff in your homedir you can have a website at no cost.
There are plenty of usecases which involve exposing ssh to the public internet.
And when it comes to raw vulnerabilities, ssh has had vastly less than stuff like apache httpd, which powers wordpress sites everywhere but has had so many path traversal and RCE vulns over the years.
We’re in selfhosted. If you have to bring up use cases that are in no way relevant to 99% of self hosters to justify your argument, you don’t have an argument.
I don’t agree about the point concerning cost. You have additional training, update, maintenance and config burden. This on top of the burdon of using the VPN on top of ssh.
This is the selfhosted community; Who are you training? In most cases there’s literally only one person who would ever need SSH access to this server. Maybe two or three in a tiny handful of cases, but anyone who can’t figure out Netbird in 30 seconds absolutely should not be accessing anything via SSH.
And you’ve clearly never used Netbird, Tailscale, or any similar service, if you think that update, maintenance and config constitute any kind of meaningful burden, especially for something as simple as remote access to a VPS.
And why exactly is that more secure?
Main reason is that if you don’t already have the right key, VPN doesn’t even respond, it’s just a black hole where all packets get dropped. SSH on the other hand will respond whether or not you have a password or a key, which lets the attacker know that there’s something there listening.
That’s not to say SSH is insecure, I think it’s fine to expose once you take some basic steps to lock it down, just answering the question.
I don’t know about Netbird specifically, but adding a VPN does a few things:
- a port scan of your VPS/router won’t show an SSH or VPN port active - Wireguard only acknowledges packets if your key is valid (massively more useful than just changing the port)
- compromising both a VPN and SSH is difficult, you’d have to chain exploits together
- if your VPN is hosted by a separate service (e.g. something like Tailscale), it will be very unlikely to share vulnerabilities with your hosted SSH server
Feel free to argue with facts. Hardening systems is my job.
And mine. Clearly one of us is better at it.
A few replies here give the correct advice. Others are just way off.
To those of you who wrote anything else than “disable passwords, use key based login only and you’re good” - please spend more time learning the subject before offering up advice to others.
(fail2ban is nice to run in addition, I do so myself, but it’s more for to stop wasting resources than having to do with security since no one is bruteforcing keys)
There’s more to it than that.
I recommend geoip blocking anything outside of your expected operating regions in addition to using key-based logins.
iptables
operates at a lower level in the network stack than SSH, so the vulnerability surface is a lot lower, and blocking before something actually looks at the packets cleans up the logs. This is huge because it makes it a lot more obvious when there’s a legitimate attack.Cover yourself with layers:
- block obviously bad packets at the firewall level
- eliminate insecure modes of login (only allow key-based login)
- something like fail2ban to ban the few who make it through 1 & 2
- use a secure root password so if someone does get in, they’re less likely to get root access
- have your services run as non-privileged users to limit issues if something gets compromised
If you only do one thing, it should be only allowing key-based logins. If you do two, run SSH on a non-standard port or set up geoip blocking (second is more work, but a lot more effective).
Still no. Here’s the reasoning: A well known SSHd is the most secure codebase you’ll find out there. With key-based login only, it’s not possible to brute force entry. Thus, changing port or running fail2ban doesn’t add anything to the security of your system, it just gets rid of bot login log entries and some - very minimal - resource usage.
If there’s a public SSHd exploit out, attackers will portscan and and find your SSHd anyway. If there’s a 0-day out it’s the same.
(your points 4 and 5 are outside the scope of the SSH discussion)
It’s also one of the biggest targets for attack. Here’s a somewhat recent CVE and here is another. Staying on top of security patches is absolutely critical, and many don’t do that.
The best security practice is to layer your protections.
your points 4 and 5 are outside the scope of the SSH discussion
They’re not about SSH, sure, but they are relevant to securing a system to remote access. Always assume your security infra will be compromised and plan accordingly. Generally speaking, the more layers, the better.