Resque to the rescue. But a gotcha: Don't forget to refresh those ActiveRecord connections

Offline queuing has been a huge issue with sites that grow to scale. I know it has been for Posterous. There are a billion options, and its totally opaque which ones are good for what.

Resque was recently open sourced by Chris Wanstrath (@defunkt) and friends at our favorite place to store code. It runs against redis, has amazing logging and forking, and is designed to be resilient under heavy load.

We recently implemented Resque for a bunch of processes around data denormalization (for your subscriptions and notifications), moving away from Workling/Starling completely. It fits perfectly, and gives us great transparency into both the queue and what happened.

After about a day, we started noticing a lot of errors showing up in the log saying Mysql::Error: MySQL server has gone away. Wild. Turns out Resque doesn't do anything special with ActiveRecord, so even though you might set up the parent proc to have your Rails environment running, there's nothing that takes care of those procs the way ActionController does.

Simple fix: We make all our tasks subclass our BaseTask, and instead of implementing self.perform(), we implement self.perform_delegate(), which our BaseTask wraps. And we call verify_active_connections! which is your friend if you want your connections to automatically reconnect if they die.




  class BaseTask
    
    def self.perform(*args)
      ActiveRecord::Base.verify_active_connections!
      self.perform_delegate(*args)
    end
    
    def self.perform_delegate(*args) # override this
    end
    
  end


Other than this little nit, we've found Resque to be phenomenal. Recommended.

views