Automatically blacklist SSH hackers

sshblack -- Notes and Details



17 Nov 2007: In response to a vulnerability pointed out by one of our users, Tim Green, we've made a very minor change, releasing Version 2.8.1. Please let me know about any issues with this release. See the README file for details. In short, sshblack now looks for the last IP address in the log line. This may cause problems with some *nix and ssh variants although we haven't seen any issues yet. There are no capability changes/enhancements with this release.

NOTE: IF YOU ARE USING SSHBLACK FOR MONITORING APACHE OR NGINX LOG FILES, please use Version 2.8 (not Version 2.8.1) available on the downloads page.


11 May 2005: The folks at MIT have released a paper regarding an SSH Worm study which was sponsored by the U.S. Air Force. It is very enlightening and worth a read. They also offer some countermeasures. The project page gives more information.


I have included some utility scripts in the tarball because people were asking me about these. I suggest you figure out what these do (and tweak them to meet your needs) before blindly executing them.



Some Notes on Upgrading

If you are upgrading to a new version of sshblack there is very little you actually need to do except kill the old version and start up the new script. Version 1 didn't use a cache file but I doubt there is anyone out there still using version 1! Even if you are, that won't cause a problem, the latest version will simply create a cache file and start working. If you are still interested in tinkering during an upgrade, read on.

Problems tend to crop up when people mess with the iptables configuration while they are upgrading. For the most part, there is no reason to do this. One situation where you might want to do this is if you are changing the $ADDRULE or $DELRULE, you may want to flush the iptables entries from the old rules. For example, if you are upgrading from 2.7 to 2.8 and decide to use -j REJECT instead of the -j DROP you were using in 2.7, you may want to flush any existing -j DROP rules. You can do this with the iptables -F BLACKLIST command, assuming your chain is called BLACKLIST. Other than that, don't worry about existing rules in the BLACKLIST chain. The new version of sshblack will work with them just like the old version did.

Nothing has changed in the format of the cache file since it was first introduced in Version 2.0. Version 2.8 will read a version 2.7 cache file just fine.

A general rule of thumb is: If you manually modify iptables, you should likewise modify the cache file and vice versa. Or delete/clear both.

Usually any errors that do show up in your sshblack log are harmless to the operation of sshblack. Tinkering with the iptables configuration or the cache file will commonly result in sshblack trying to delete iptables rules that don't exist. sshblack will continue to run even if these errors do occur. However, if you delete the entire iptables chain that sshblack is trying to use, obviously sshblack will not be able to add rules to a chain that doesn't exist.


Maintaining things with cron, crontab and iptables-save

Some have suggested that sshblack should fully maintain the iptables configuration it works with, including saving, checking and restoring chains and rules. I have chosen not to do this for several reasons which are beyond the scope of this section. However, here are some pointers that allow you do do these kind of operations yourself.

If you'd like to manage sshblack like any other background process, see The Configuration Page for some user-supplied information on managing sshblack with things like chkconfig. There is even an init script.

Actually, starting sshblack after a reboot can be as simple as placing the full path and file name in your /etc/rc.d/rc.local file (or whatever directory/file your OS uses for custom start-up scripts). This will start sshblack very quickly after your machine boots.

Saving the iptables configuration can be important because if the custom chain sshblack uses is not restored after a reboot, obviously sshblack won't be able to add/delete rules for a non-existent chain. If your machine supports it, saving and restoring iptables configs can be done easily using iptables-save and iptables-restore.

Execution of the iptables-save command can be done in the root crontab or it can be placed in the /etc/cron.daily directory. You can of course do this save every hour if you like by placing the shell file in the /etc/cron.hourly directory. Here is an example of an iptables-save script:

#!/bin/sh

# Save iptables configuration to /etc/sysconfig/iptables.1
/sbin/iptables-save -c > /etc/sysconfig/iptables.1

If you'd like to use crontab to do this same thing you can execute the following command:

[root@stinky root]# crontab -e

You will then see your crontab configuration (likely opened in vi). Simply place the following command into your crontab file and save it:

mailto = "root"

25 * * * * /sbin/iptables-save -c > /etc/sysconfig/iptables.1

Now, how do we pull this information back into iptables in the event of a reboot? Simple, just go back to your /etc/rc.d/rc.local and add in the iptables-restore command [Be sure to place this line BEFORE the line that starts sshblack in your rc.local file]. This will pull that saved information back in to iptables.

/sbin/iptables-restore -c < /etc/sysconfig/iptables.1

Note that iptables-restore will not only restore the rules that sshblack has added, it will restore any custom chains also.


Prerequisites




A Few Notes (aka Rants)

Let me make a couple points in response to some comments I've seen regarding this script and active blacklisting in general.

1. My passwords are secure. Who cares if someone is hacking at them?

In your case and, perchance, my case, this may be true. If you are certain your passwords will not fall to a dictionary, brute force or intelligent attack and you see no other need to thwart hacking attempts, I say, "Good on ya!" Move on now and don't worry about sshblack (and don't bother to check your logs because you won't be able to find anything buried in those 2000 lines of failed ssh logins). However, if you are administering a system where you might have 100 or 1000 users and have no idea how strong all those passwords are, you might want to protect yourself. Ask anyone who administers a large system and they will tell you that, despite best efforts to enforce strong passwords, there will always be someone who uses "password" or "temp". Fortuntately many system have integrated password screening that will not allow users to set weak passwords. Many do not. Also, even smart people make stupid mistakes. Perhaps you are setting up a test account or installing a new piece of code that requires a user account. You forgot to remove that account when you were done testing or forgot to disable logins for that user. Many machines run by smart people have been compromised when these kind of mistakes are exploited. Don't worry, sshblack has your back.

2. It would take nine years to hack even a five character password! That's safe enough.

Your math is flawed. First, this assumes a brute force attack which may or may not be a good assumption. Secondly, you are assuming that you are only going to attack on one socket at a time and wait for the ssh daemon to time out or fail each time. That's (usually) not the way it works. Most operating systems running ssh have the ability to open more than one socket/port/thread. That means your ssh daemon can be serving 10, 20 or 1000 log in attempts all at the same time. It's extremely simple to fork a password hacker to open multiple login attempts at the same time. Do the math on that. Fortunately most of the Linux ssh code allows you to adjust the number of threads (either logged-in or initializing) in a config file. Some systems do not have this limiting feature. Yet even if you limit this to five simultaneous login prompts, you are still not safe. See the next paragraph. I would also point out that MANY people use sshblack to monitor/control things other than ssh daemons so it's utility goes well beyond failed password attempts.

3. I've limited the number of threads in my ssh daemon and use strong passwords. It would take someone over 12,000 years to hack me!

They don't have to be successful at cracking a password to hurt you. They can use a multi-thread attack to tie up all of your ssh sockets or at least run up your process count and suck up bandwidth. Obviously the biggest threat here is denial of service. If they have all your ssh threads tied up with their hacking attempts, how are you going to log in? This is why it is so important to stop these attacks at the most primary layer. In the case of sshblack this is done at the IP layer. By the time you are checking passwords or access databases, its too late.

4. Someone could spoof my address and get me blacklisted from my own machine!

There are several SIGNIFICANT issues with this argument. First, to even attempt this denial of service, someone would have to assume that you are actually running a script such as this. That's knowledge that your average attacker isn't going to have and knowledge the script kiddie is never going to slow down to acquire. I would also hope that most people have an upstream router that should be ignoring local-net spoofed packets and not forwarding them on to your SSH host. As for spoofing your remote address, such as your home DSL, yes, I suppose if someone was so determined and had a HUGE amount of a priori knowledge they could launch such an attack. Secondly, let's assume that someone has all this knowledge, maybe a disgrunteled ex-employee. Lauching a blind TCP spoof attack is not impossible but also not that likely. Remember, we are not talking about a smurf attack or anything like that. They have to get the ssh daemon to believe that it is talking to a routed host when it really isn't. What are the chances someone is going to go to all that trouble just for DoS? There are much more efficient DoS attacks they could use. Welcome to the Internet. Lastly, and most importantly, a whitelist function is included with sshblack so you should be able to whitelist any host or network you like and the rest of this discussion becomes nonsequitur. If you use the WHITELIST this problem should be significantly minimized.

5. The attacker could use a botnet or spoof addresses until my iptables fills up with thousands of entries!

Well, they could except for the fact that sshblack shuts itself down for 24 hours if it detects such an attack. You can limit the total number of blacklist entries to 50 or 200 or 10,000. Whatever you're comfortable with. And when it shuts down, the hacking attempts go on unabated. But you are the same folks that said nobody should be using a script like this in the first place so that's fine with you. You don't check logs or worry about the machine anyway so what's one more DoS attack, right? They've probably tied up all your ssh sockets so you can't log in anyway. This whole argument falls to some of the same fallacies that the self-blacklisting-blind-spoofing DoS threat does.

6. Why would you care?

I don't know how I can make this any more clear: If you never look at your logs and/or don't care about hundreds or even (yes) thousands of lines of hacking attempts in your logs, do not bother with this script. I tend to do everything I can to limit my exposure to viruses, trojans, script kiddies, spammers and the like. This is one small tool I use to limit one aspect of that exposure. I use blacklists on mail servers too. If that offends your sense of society, please don't use them. My passwords are strong, my software is up to date. It is not that I think these attempts are going to be successful at cracking my SSH daemon. I just get tired of seeing all the garbage. I trust the deadbolt on my front door too. That doesn't mean I want guys lining up in the street trying random house keys.

7. Locking out users after X many failed attempts is a bad idea.

Really? I'm the first one who's ever thought of this? Wow! I think if you look, you'll find dozens of operating systems, applications and servers that do exactly this. SSH is not for your grandma. If a user can't manage to enter a password correctly after four (or more) attempts, maybe they should take up a new profession or avocation.



Other Options

The script works very well as it is however, many of the discussion boards have also presented some other options. I use a few of these myself. Only a couple of these will put a complete stop to the problem, but they have their associated drawbacks. Quite a few of these are simply good general security measures that should be done even if you don't have an SSH attack problem.

	ListenAddress 192.168.209.10:222
	
	PermitRootLogin no
	
	uucp:x:56:15:uucp:/var/spool/uucp:/sbin/nologin
	games:x:58:100:games:/usr/games:/sbin/nologin
	gopher:x:99:2:gopher:/var/gopher:/sbin/nologin
	bubba:x:600:611:Bubba Gump:/home/bubba:/sbin/nologin
	
	LoginGraceTime 20
	MaxStartups 1
	
mail
Copyright 2006 Pettingers.org

Vectors at

pettingers.org