Cleaning Up Redis Cloud

When your project needs to include a background message system, Redis certainly comes to your mind. As a software developer, I already used it a lot on the application level. However, the Redis database itself was an unknown territory for me, and the need to face it came up one day.

This post will be an account of the steps I did to perform a bulk delete from a certain Redis key that was occupying too much storage space. The action looks incredibly simple and certainly is for someone who already performed it once, but I find it difficult to gather material to do this the first time I needed it.

The problem

A Heroku application was using a Redis cloud solution, and the thing was going messier gradually. The data size was increasing and always after a few months, an upgrade plan was necessary. It came up that the next one would be too expensive, so it was decided to give it a look.

The cause was pretty straightforward: we stored user sessions on Redis, and because of a typo in the code, they weren’t marked to expire, so it was necessary to clean these sessions up.

Redis allows you to scan for those keys, however, this action is slow and things can get worse accessing your Redis cloud database because of the server latency. So I ended up whit a task impossible to be done directly on the server in a reasonable time.

The solution

Do the scan in your local machine. First and foremost, you need to have `redis` and `redis-cli` (it comes packed with `redis-server` already) installed. My suggestion is to download the last stable version of Redis with wget http://download.redis.io/redis-stable.tar.gz and follow this great tutorial to complete your installation and setup.

The strategy was based on a few steps:

1To avoid the latency problems, download the last backup from the Redis cloud database and restore it to your local machine.

With your backup in your hands, be sure to stop the Redis server first: sudo service redis_6379 stop .

Rename your current dump.rdb file to something different: mv /var/lib/redis/6379/dump.rdb /var/lib/redis/6379/dump.rdb.old.

Copy your backup to the same folder above: cp /backup/dump.rdb /var/lib/redis/6379/ .

Set it to be owned by the Redis user/group: chows redis:redis /var/lib/redis/6379/dump.rdb.

Finally, start your server again: sudo service redis_6379 start .

(Of course, service names and directories can change slightly according to the Redis installation you made).

2 Perform the scan to build a text file that will contain all the key names, one per line: redis-cli --scan my_key* > keys_to_delete.txt. The * is a special character and in this example, the scan will return every key that matches my_key at the beginning of the name.

3 Delete all the keys running many Redis del commands. The del command can fail if we inform a number too big of keys at once, so we run the script below:

cat keys_to_delete.txt | xargs -n 1000 redis-cli -h redis.us.cloud.redislabs.com -p 9999 -a my_password del

xargs allow us to get the whole content thrown by cat and perform in batches of 1000 rows each time the Redis del command in our Redis cloud database through redis-cli.

And this is it! The process can still be slow according to the size of your database and the number of keys to delete, but that way we bypass the latency problem that was preventing us to search by the keys to be deleted.