Why prefer Redis over others?
Most of the developers nowadays would have heard about Redis. Redis is one of the best open source in-memory NoSQL database currently available in the market. It offers a lot of good to have features for frontend as well as backend services like key-value lookup, queues, hashes, hyperlog (etc).
It’s been 2 years since we started Pepipost. We developed the Pepipost product with a combination of PHP-MySQL and Perl. The initial few months were smooth. When we started sending larger sending volumes, we began to experience performance issues with some components. So we dug deeper to find the problems and solutions for it. Our conclusion? Get memory cache to speed up as most queries for blocking were at the MySQL level. We had used Memcached in the past, so we started to search based on that and the fact that it is best in this category. While we were hunting for the solution, one of our developers came up with the suggestion of using Redis, which I misheard as RADIUS. Being from an Internet service provider background, I had used this Radius and I thought "How can we use that as a database?"
After having an extended discussion , we went through all docs & benchmarks of Redis & RabbitMQ, finalizing on Redis, as it had many features. We had no use for some of it at that particular time, but we prefer to have lots of features in the products we buy, even if we are not using them quite yet. Always looking forward!
We added one Redis server which started to cache all the client configs inside it, which reduced the config queries to MySQL by 50%. Once this was in place, we broke down our back end components in micro services and managed that using Redis Queues. When we started we had 10 queues; currently, it’s in the 100's, each serving a separate purpose.
Redis Usage in Pepipost.
Here at Pepipost, we use Redis in multiple scenarios, right from accepting emails from HTTP API or SMTP and storing them on Redis queue, then generating the final email and delivery of email. Each email triggers multiple events as it progresses. We use real-time dashboards using Redis to show the number of emails that have been sent today, with the number of opens, clicks, bounces on a real-time basis. We have priority queues, job queues, real-time dashboards, retry failed events after a particular time interval, etc. using Redis. At any given time, our Redis servers process around 40K requests per second.
Check out some stats!
-----------------------------------------[Pepipost@~]$ redis-cli info stats
Gradually we moved all our dependency from MySQL to Redis, so that even if MySQL is down, there is Zero impact on Pepipost from acceptance of email to delivery. This has helped us to maintain 99.99% up-time to our customers at all times.
At times we have hit bottlenecks in Redis also. Currently, we use multiple Redis servers to handle the enormous load of the email volume we are delivering. Our engineering team always looks for next bottleneck that is impacting the overall performance, and optimizes it. Our team is doing POC on NSQ , which as per their blog can give up to 400K request per second in single node & 800K request per second in 3 node cluster. We are expecting to get a very nice jump in our performance using this.
How you can use Redis
Different ways in which you can use Redis.
- Key-Value stores - To store individual configs items.
- Hashes - Can use this to store data which can be huge key-value lookups & needs to be broken down. For example, you have a blacklist database with millions of records that need to be looked up very frequently. Using Redis Key-Value is not the best method as millions of keys will bring the performance down. Fortunately, there are other alternatives like Constant Database (TinyCDB) for this use case.
- Counters & Hyperlogs - Best to build real-time dashboards & leaderboards.
- Sorted Sets - You can use this for triggering events at a particular time, eg drip campaigns, automation of events.
- Bitmaps - Good to build cohorts and other analysis. Learn more
There are much more types of features available in Redis which you can use, visit https://redis.io/commands