As System Administrator, you are familiar with security. You have servers that provide internet services to the public. You make a daily check on the logfiles when the busyness of your day allows. Mostly these controls are used to check that your services are operating correctly. These days secuirty checks have become even more important.
At one of my clients I heard that “IMAP and SMTP TLS are closed.” The Systems Administrator at this client had noticed that a large number of password checks were being attempted. “It is only a matter of time before a password is hacked!” I think that there is a better alternative than a complete closure of secure (e.g. TLS encrypted) services where hackers can indeed access passwords from. An elegant tool for this is fail2ban.
3DN exclusively uses Linux servers. On Linux servers many firewalls are often supplemented by help from iptables. Iptables is a method by which to inform Linux kernel which network packets from the OS layer may be forwarded to the application layer. When you use the kernel with the iptables for example, and you type:
iptables -A INPUT -i lo -j ACCEPT
You are telling the kernel that:
- All the network packets on interface (-i) lo, the loopback interface, that come through will be accepted and will be passed to the application layer or the kernel itself will handle the packet.
Possibly you can already immediately see the complications. Iptables commands are often difficult to read. The average layman isn’t going to be able to do this and even some lesser sysadmins often have problems.
There are all sorts of applications of varying usefullness that make firewall protection simpler to use. The application UFW (Ubuntu Firewall), for example. With UFW it becomes much easier to both read and implement firewall tables:
ufw allow 53/udp
The above example shows how simply UDP traffic for port 53 (DNS) can be allowed. UFW, like many other tools, works underwater with iptables but has little influence of the command structure.
Static or Dynamic
Now we come to the heart of this article. The making of a static firewall table can be easily learned and there are clear methods by which senior sysadmins can refine their firewall details. Even more, the internet has a plentiful collection of users of your services.
Less experienced firewall administrators, when observing lots of hacking attempts, can block an IP address. So, too, did a network administrator from former client on noticing heavy internet activity on TCP port 3127. In the past there was often a policy that “Everything stays open until we notice misuse. Then we close it.” The sysadmins would then immediately close TCP port 3127. This had the effect that the already infected local PCs would try to gain access via TCP port 3128. By chance was this a well-known TCP port from a known service – the Squid cache. The Squid cache was at that time frequently used as a local caching in order to look at slow internet connections.
From the above we can see that the closure of certain ports don’t always achieve the desired result. The writers of the virus in the example just given knew the mindset of the average firewall administrators and had cheerfully taken advantage of this fact. As a result, the Squid cache became practically unusable because it was so overloaded. In essense it was handily made into a so-called DOS (Denial of Service) attack. An attack created whereby the infected PC’s from the client themselves created the DOS on the Squid server.
A firewall administartor these days needs to have a more dynamic character, who realizes that there are hundreds of thousands of hackers alive on the internet. These hackers vary in expertise from “script kiddies,” people who download a script from the internet and then go hacking, to experts that have an insight in the mindset of security admins and are quite flexible in how they react to it.
There are also sysadmins who have a pretty good idea how hackers think. The pattern we described eariler:
You make a daily check on the logfiles when the busyness of your day allows. Mostly these controls are used to check that your services are operating correctly. These days secuirty checks have become even more important.
containes many “gotchas” from the modern sysadmin. The controlling of logfiles with ever-growing servers is always mind-numbing work and often practically impossible, seeing as how management often does not give time for such work. Although businesses are aware of the importance of security, they are typically unprepared to follow up on this knowledge.
System Administrators can therefore these days make use of tools that read the logfiles for them. An excellent example of this is the logscan tool. Logscan aggregates frequently repeated messages in each of the logfiles into a readable whole. Logwatch then sends an insightful email to the sysadmin such as this:
sshd: Authentication Failures: root (18.104.22.168): 9 Time(s) root (22.214.171.124): 7 Time(s) root (126.96.36.199): 6 Time(s) root (188.8.131.52): 6 Time(s) root (184.108.40.206): 6 Time(s) root (220.127.116.11): 6 Time(s) root (18.104.22.168): 6 Time(s)
Above you can see a few IP addresses which have attempted to guess the password of a user. When an organization has a strong password policy this is not a problem. Random attempts to guess the password will never result in a correct password guess.
If your organization does not have a strong password policy, the probe will continue its attempts over and over until finally the password is found. The real question is, if you have a weak password policy do you decide to close the SSH port, or does your organization actually have a pressing need to enforce a strong password policy upon its users?
Even with an excellent password policy it can be irritating when your SSH service becomes more and more well-known, sometimes in distributed databases of hackers cooperating in teams, making more and more IP addresses scan your service. A simple login attempts results in the case of SSH to a decryption process that’s reasonable expensive in terms of CPU cycles. When tens of thousands of IP addresses do this per day, you will start to notice you servers performance suffering.
At this point it’s time to start using fail2ban.
Fail2ban, like logwatch, can watch all sorts of logfiles and doesn’t do this periodically but continuously. The number of logfiles fail2ban can monitor at the same time is nearly limitless, the number of methods by which these logfiles can be watches extensive.
Fail2ban can apply a filter and actions to a logfile when the filter matches. Basically it boils down to:
- Open a logfile and keep watching it as long as the system is up and running.
- Check if the logfile additions meet a certain regular expression criterium.
- If the above regular expression evaluates to true, take action against the offending IP. In many cases the first actions will be a temporary iptables block to slow down the attacker.
In the next chapters we’ll demonstrate a few use cases of popular services after giving a short explanation on the installation and configuration of fail2bna.
We install Fail2ban on a Debian 9 system. Fail2ban is part of the standard Debian repositories and can be installed without any problem with the apt-get command:
root@gamma:~# apt-get install fail2ban
At the time of writing this article this will install version 0.9.6 of fail2ban.
We configure fail2ban quite simply by:
root@gamma:~# cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
This should be the very first thing you do; it’s the copying of the configuration file that may be overwritten by updates to a file that’s safe from overwriting by accident.
For use case 1 we then edit jail.local and adjust the SSH sectie the following way:
[sshd] enabled = true port = ssh logpath = %(sshd_log)s backend = %(sshd_backend)s
The ‘enabled = true‘ line makes sure that at restarting fail2ban, the SSH jail will be activated. Other jails used in Use case 2 and Use case 3 will be explained in the relevant chapters.
Use Case 1: Fail2ban for protecting SSH
SSH, or secure shell, has an advantage and a disadvantage. It’s a classical example of a secure service. Many things related to security have been carefully considered. It’s for example possible, and desirable, to make root login impossible. It’s also possible to make logging in with a password completely impossible when we have generated a so called key. The good security design of SSH however has also cause SSH to be opened to the internet because ‘it’s safe anyway’.
The above however has also caused SSH to be a frequent target of brute force attacks. During a brute force attack, an attacker attempts many times to guess the password. Often there’s more intelligence behind it than it would appear like at first looks.
Due to the huge number of brute force attacks, SSH is by far the largest target of attackers. 95% of all attacks appears to be directed against SSH. Partially because of this, securing SSH with fail2ban is very simple.