Drupal Critical Update SA-CORE-2014-005 - How to Fix an Infected Site

The issuing of the security advisory SA-CORE-2014-005 - Drupal core - SQL injection on 15th October, and designated the status of Critical sent shock waves through the Drupal community. The inference being that unless the security fix was applied within seven hours of the advisory then a Drupal site owner should expect their site to be subjected to attack.

I look after many sites and quickly applied the fix to them. However I couldn't react within the seven hours - in fact I didn't even hear about the issue so quickly. Therefore it couldn't be considered a surprise when I discovered one of the sites I looked after had become infected.

The security advisory is suitably vague about the consequences of any attack and so I can only supply a fix for the problems I experienced on my site. Any other infected site may have different symptoms and consequences to mine obviously, and your mileage may vary.

The symptom I experienced was numerous email non delivery reports flooding my email client. All the original messages were emanating from my machine, and being created and issued through Apache Web Server meaning a script on one of my sites was generating the messages through an infected script. The subsequent attempt of external delivery was resulting the NDRs coming straight back to my email client.

Mutt NDRTo diagnose the issue, I spun up the email client mutt - which should be familiar with most dyed-in-the-wool Linux Engineers. Other clients will be able to provide the same information if you are not familiar or don't have mutt installed. You need to look at the header information to determine the provenance of the message so in mutt I hit return over one of the messages then hit 'h' for header information.

If you look carefully you will see under the header X-PHP_ORIGINATING-SCRIPT there is reference to db60.php and a particular line number. Bingo! We now know what generated the emails.

So to find the script simply issue a find command.

openSUSE-123-64-minimal:/srv/www/htdocs/tube # find . | grep db60.php
openSUSE-123-64-minimal:/srv/www/htdocs/tube # cd modules/translation
openSUSE-123-64-minimal:/srv/www/htdocs/tube/modules/translation # ls -las
total 88
4 drwxrwxr-x  3 wwwrun www  4096 Nov 26 04:53 .
4 drwxrwxr-x 42 wwwrun www  4096 Nov 19 20:24 ..
20 -rw-r--r--  1 wwwrun www 19175 Nov 19 20:24 db60.php
4 drwxrwxr-x  2 wwwrun www  4096 Nov 19 20:24 tests
4 -rw-rw-r--  1 wwwrun www   322 Nov 19 20:38 translation.info
24 -rw-rw-r--  1 wwwrun www 23389 Nov 19 20:24 translation.module
4 -rw-rw-r--  1 wwwrun www  3278 Nov 19 20:24 translation.pages.inc
24 -rw-rw-r--  1 wwwrun www 22087 Nov 19 20:24 translation.test
openSUSE-123-64-minimal:/srv/www/htdocs/tube/modules/translation #

db60.phpThe file was copied to the /tmp directory and inspected. The file is obfuscated PHP and the eval command with the email generating payload is at the end.

Next step is to determine the IP address responsible for executing this script. This can be achieved by issuing the following:

openSUSE-123-64-minimal:/srv/www/htdocs/tube/modules/translation # grep db60.php /var/log/apache2/access_log | awk '{print $1}' | sort -n | uniq -c  | sort -n

A whois search indicates this is a Russian address. Wherever it originates from, it needs to be blocked immediately. The best solution would be to block at it the firewall method. Since I wanted a solution then and there I blocked it first against the website by adding the address to the .htaccess file
openSUSE-123-64-minimal:/srv/www/htdocs/tube # tail -3 .htaccess
order allow,deny
deny from
allow from all

Now I wanted to determine how this mb60.php was being generated. I navigated to the document root of the website and the answer was apparent immediately. There should be no general.php in the root directory of a Drupal site.
openSUSE-123-64-minimal:/srv/www/htdocs/tube # ls -las *.php
8 -rw-rw-r-- 1 wwwrun www  6604 Nov 19 20:24 authorize.php
4 -rw-rw-r-- 1 wwwrun www   720 Nov 19 20:24 cron.php
24 -rw-rw-r-- 1 wwwrun www 23025 Dec 11  2013 general.php
4 -rw-rw-r-- 1 wwwrun www   529 Nov 19 20:24 index.php
4 -rw-rw-r-- 1 wwwrun www   703 Nov 19 20:24 install.php
20 -rw-rw-r-- 1 wwwrun www 19986 Nov 19 20:24 update.php
4 -rw-rw-r-- 1 wwwrun www   417 Nov 19 20:24 xmlrpc.php

db60.phpYet more obfuscated PHP code which I immediately moved away from document root into the temporary directory.

I then decided to check the root directories of all my other Drupal 7 sites, and lo and behold I spied another rogue file, this time entitled css.php in one of them. I simple diff command proved it was a copy of the general.php file.

openSUSE-123-64-minimal:/srv/www/htdocs/sdsb # ls -las *.php
8 -rw-r--r-- 1   6226 6226  6604 Nov 19 20:24 authorize.php
4 -rw-r--r-- 1   6226 6226   720 Nov 19 20:24 cron.php
24 -rw-r--r-- 1 wwwrun www  23025 Apr 27  2014 css.php
4 -rw-r--r-- 1   6226 6226   529 Nov 19 20:24 index.php
4 -rw-r--r-- 1   6226 6226   703 Nov 19 20:24 install.php
20 -rw-r--r-- 1   6226 6226 19986 Nov 19 20:24 update.php
4 -rw-r--r-- 1   6226 6226   417 Nov 19 20:24 xmlrpc.php
openSUSE-123-64-minimal:/srv/www/htdocs/sdsb # less css.php
openSUSE-123-64-minimal:/srv/www/htdocs/sdsb # diff css.php  /tmp/general.php

So the lesson is to check that your Drupal 7 installation has exactly the correct number and correctly named php scripts.

Next step is to remove all the messages in the mail queue waiting for delivery to be retried. I use exim for my MTA so my command is as follows:

openSUSE-123-64-minimal:~ # exim -bp
25m  1.1K 1XtXXl-0003eD-98 <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXm-0003f4-Fl <wwwrun@calsync.co.uk>

25m  1.0K 1XtXXn-0003fS-2n <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXn-0003fZ-6J <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXn-0003ft-Ma <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXo-0003gB-5y <wwwrun@calsync.co.uk>

25m  1.0K 1XtXXo-0003gG-Ay <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXo-0003gL-Eh <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXo-0003gQ-Ip <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXo-0003gi-US <wwwrun@calsync.co.uk>

25m  1.1K 1XtXXp-0003gn-1v <wwwrun@calsync.co.uk>
25m  1.1K 1XtXXp-0003gt-6t <wwwrun@calsync.co.uk>                                                  
openSUSE-123-64-minimal:~ #

To delete using exim use:

openSUSE-123-64-minimal:/srv/www/htdocs/tube # exim -bp | awk '/^ *[0-9]+[mhd]/{print "exim -Mrm " $3}' | bash
Message 1XtXXl-0003eD-98 has been removed
Message 1XtXXn-0003fZ-6J has been removed
Message 1XtXXn-0003ft-Ma has been removed
Message 1XtXXo-0003gB-5y has been removed
Message 1XtXXo-0003gG-Ay has been removed
Message 1XtXXo-0003gL-Eh has been removed
Message 1XtXXo-0003gQ-Ip has been removed
Message 1XtXXo-0003gi-US has been removed
Message 1XtXXp-0003gn-1v has been removed
Message 1XtXXp-0003gt-6t has been removed
openSUSE-123-64-minimal:/srv/www/htdocs/tube #

My final (and probably worthless) step was to create a cron job to automatically delete db60.php should it reappear, although I think I'll remove this entry in a couple of weeks since I believe the entire problem is now fixed.

openSUSE-123-64-minimal:/srv/www/htdocs # cd /var/spool/cron/tabs/
openSUSE-123-64-minimal:/var/spool/cron/tabs # tail -2 root
# Delete hacker script
* * * * * rm /srv/www/htdocs/tube/modules/translation/db60.php > /dev/null 2>&1
openSUSE-123-64-minimal:/var/spool/cron/tabs #