Resque to the rescue. But a gotcha: Don't forget to refresh those ActiveRecord connections

Offline queuing has been a huge issue with sites that grow to scale. I know it has been for Posterous. There are a billion options, and its totally opaque which ones are good for what.

Resque was recently open sourced by Chris Wanstrath (@defunkt) and friends at our favorite place to store code. It runs against redis, has amazing logging and forking, and is designed to be resilient under heavy load.

We recently implemented Resque for a bunch of processes around data denormalization (for your subscriptions and notifications), moving away from Workling/Starling completely. It fits perfectly, and gives us great transparency into both the queue and what happened.

After about a day, we started noticing a lot of errors showing up in the log saying Mysql::Error: MySQL server has gone away. Wild. Turns out Resque doesn't do anything special with ActiveRecord, so even though you might set up the parent proc to have your Rails environment running, there's nothing that takes care of those procs the way ActionController does.

Simple fix: We make all our tasks subclass our BaseTask, and instead of implementing self.perform(), we implement self.perform_delegate(), which our BaseTask wraps. And we call verify_active_connections! which is your friend if you want your connections to automatically reconnect if they die.




  class BaseTask
    
    def self.perform(*args)
      ActiveRecord::Base.verify_active_connections!
      self.perform_delegate(*args)
    end
    
    def self.perform_delegate(*args) # override this
    end
    
  end


Other than this little nit, we've found Resque to be phenomenal. Recommended.

Why the process-monitoring ruby 'god' broke

We were having all kinds of weird problems with god, and had to resort to weird hacky solutions to try to make it work. Turns out, there's a reason for it. If you try to do all sorts of shell-like actions in the commands for start/stop/restart, god ends up throwing it all in a shell process, e.g. sh -c your_cmd.

But then god gets the pid to the SHELL process and not your_cmd. And when you want to restart, it ends up killing the shell command and leaving you with zombie, orphaned, totally untracked processes that you have to kill -9 manually. Gruesome.

So in short, this stuff you should AVOID:


w.start = "cd #{RAILS_ROOT} && QUEUE=#{name} rake do:mytask"

This is OK:


w.dir = RAILS_ROOT
w.env = {'QUEUE'=> name}
w.start = "rake do:mytask"

And knowing is half the battle!

EDIT: You could also think of using Bluepill, which is God without the insane memory leak (which apparently is fixed in some hacker's dev branch!) It's in use at Serious Business, and they are one hardcore operation. http://github.com/arya/bluepill

Which acts_as_solr should I use? Mattmatt's acts_as_solr, naturally.

script/plugin install git://github.com/mattmatt/acts_as_solr.git

A note about the acts_as_solr codebase: it all started with an innocent hack that I posted to the solr-user list. It got picked up [editor 3/18/09: respectfully added a special mention of Thiago Jackiw] by Thiago and he turned it  into a serious general purpose ActiveRecord modeling plugin hosted at RubyForge, and now exists as numerous git repository forks. The currently best maintained version is Mathias Meyer’s branch.

Rails plugins have an incredibly short half-life. The creators of acts_as_solr, railsfreaks.com, seem to have disappeared. Luckily, open source can be picked up by new baton carriers.

Thanks mattmatt. You rock.

Flash security policy attacks hit Facebook/MySpace... a techie look at what happened

Cross-domain policy files (crossdomain.xml) are forgivingly parsed by Flash. If an attacker can construct an HTTP request that results in the server sending back a policy file, then Flash will accept the policy file. For instance, imagine a university website that responds to a course listing request:

http://www.example.com/CourseListing?format=js&callback=<cross-domain-policy><allow-access-from%20domain="*"/></cross-domain-policy>

...with the following output:

<cross-domain-policy><allow-access-from%20domain="*"/></cross-domain-policy>() {  return {name:”English101”, desc:”Read Books”}, {name:”Computers101”, desc:”play on computers”}};

Then one could load this policy via the following ActionScript? code:

System.security.loadPolicyFile("http://www.university.edu/CourseListing?format=json&callback=<cross-domain-policy>" + "<allow-access-from%20domain=\"*\"/></cross-domain-policy>”);

This results in the Flash application having complete cross-domain access to www.example.com.

Github releases Resque. Wow, this is exactly what we wanted in Rails queueing. Much respect.

Resque is our Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.

Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.

This goes into great detail into the tradeoffs and problems with existing delayed job queueing systems. Brett was mentioning -- this post reads like a laundry list of stuff we've dealt with!

Resque is a Delayed-Job-like queue that is built on Redis instead of MySQL. Brilliant.