Rolling back a Drupal site following Drupalgeddon

There is a lot of talk in the Drupal community and media about the Drupal security vulnerability that was fixed in the Oct 15th update (7.32). If you have missed the details, here is a summary:

A highly critical security vulnerability was found in Drupal's database abstraction API. This was fixed with the release of 7.32 on Oct 15th. If you did not upgrade to Drupal 7.32, or apply the patch, by 11pm UTC on Oct 15th, your site may be compromised. Applying the patch after this will not remove any backdoors that may have already been added to your site or server.

So, if you upgraded or applied the patch immediately, or within 7 hours, you are safe (or as safe as you can be). But if you didn't, it is time to take precautionary measures. The best and most effective measure to take is to roll back everything to a state before Oct 15th 2014. And when I say everything, that includes the server not just the web files and database because backdoors could have been added anywhere.

I upgraded my clients site within 1 hour of the 7.32 release, so they are safe. But I had a couple of personal sites where I missed the 7 hour window by a few hours. Despite not seeing evidence of any hacking, I performed the rollback as a precautionary measure. What follows are my notes on how I did this.

This is what I had:

  • 3 sites running as a multisite
  • Shared hosting with WHM and cPanel
  • Database backups to Amazon S3
  • Code version controlled using Git

Prepared the clean site

I prepared the clean site on my local machine. I got the database backup that pre-dated Oct 15th from Amazon S3 and restored to the local version of the site.

Import new content

I imported any new content created on the live site after Oct 15th. If the site isn't updated that frequently, you may do this manually. Or you could export content using Node Export module. Just be sure to inspect every node for possible anything that may have been added by a hacker.

In my case, I manually recreated the new nodes. I did this by going to admin/content on both the local site (which now has the clean database from the backup) and the live site and created anything new. For each node, I changed the Authored on date to reflect the original publication date. I also made sure that the node IDs matched both the live site and the local site by creating the nodes in the correct order. This is essentially for importing comments.

Import comments

The first thing I did was moderate any new comments on the live site. Then I checked the last cid in the comment table on the local database. Following that I went to phpMyAdmin for the live site and selected all rows after that cid and exported those rows, and after that imported into my local site. There are three tables that contain comment data that need to be exported/imported: comment, field_data_comment_body, field_revision_comment_body, node_comment_statistics.

Copy files

Copy down any new images in the files directory. Be really careful here because the files directory is a common place to put dodgy files. Make sure you only download images (PNG, JPEG, GIF). Do NOT download the .htaccess file (this could have been hacked).

Backup live database and files

Backup the current live files and database. I did this to a new Amazon S3 bucket. I'm only doing this just in case I need it again in the future.

Dump local clean database

Dump the local, clean, database. This now has the new nodes and comments. I prepended the database name with "clean_" to make it clear that this is the clean database that I'll use to import on the live site.

Rebuild the server

Now it is time to wipe the server and rebuild it. In my case, this is managed by WHM. So I simply terminated the cPanel account and recreated it again. This gives me a fresh clean cPanel account to add everything to again. I spoke to the host before doing this and they agreed that it was the best course of action.

Please bear in mind that this will remove your databases, database accounts and mailboxes. My email runs through Google Apps, so that wasn't a problem.

DNS issues

There was a DNS issue when I recreated the cPanel account. A quick email to the host and they sorted that out.

Deploy the code

With the old database and files gone, it is time to deploy the files, create the databases, users and import the database.

Drupalgeddon free

Now I am safe in the knowledge that the site and server is in a state pre-dating Oct 15th with content migrated and safe from Drupalgeddon.

What if you don't have backups?

In order to roll back like I outlined above, you need database backups and version controlled code. If you don't have both of those things, it gets a whole lot messier. You have two options:

1) Throw everything away and start again

2) Check for common signs of vulnerabilities. And then keep checking on a regular basis. Here are some resources to help you with that:

Going forward

Drupalgeddon serves as a reminder to everyone to have a proper backup strategy, store your code in version control and be ready to apply security updates when they are released every Wednesday.

You can get security advisories from these places:

Comments

Nice article Blair!

I would power down (rather than wiping) the server after downloading your files - just in case you missed something. This is probably what you did but I thought I would mention this for the benefit of your readers.

Thanks for writing this Blair - it is very handy for me. I too have WHM and cPanel, so I'm just double checking with my hosting provider if I can do the same as you.
I wonder if you have any idea how, and if I need to reset the passwords of users that log in with HybridAuth social log in?
Thanks
Craig

Blair Wadman's picture

Hi Craig,

I did reset passwords for all users on the sites I rolled back, but there weren't that many. As for users who registered using HybridAuth, I don't believe that is necessary because they are authenticated by the social network. However, users with HybridAuth have a normal Drupal account, so if they can login using their Drupal password, it is probably a good idea to reset those as a precaution.

Add new comment