Resque to the rescue. But a gotcha: Don't forget to refresh those ActiveRecord connections

Offline queuing has been a huge issue with sites that grow to scale. I know it has been for Posterous. There are a billion options, and its totally opaque which ones are good for what.

Resque was recently open sourced by Chris Wanstrath (@defunkt) and friends at our favorite place to store code. It runs against redis, has amazing logging and forking, and is designed to be resilient under heavy load.

We recently implemented Resque for a bunch of processes around data denormalization (for your subscriptions and notifications), moving away from Workling/Starling completely. It fits perfectly, and gives us great transparency into both the queue and what happened.

After about a day, we started noticing a lot of errors showing up in the log saying Mysql::Error: MySQL server has gone away. Wild. Turns out Resque doesn't do anything special with ActiveRecord, so even though you might set up the parent proc to have your Rails environment running, there's nothing that takes care of those procs the way ActionController does.

Simple fix: We make all our tasks subclass our BaseTask, and instead of implementing self.perform(), we implement self.perform_delegate(), which our BaseTask wraps. And we call verify_active_connections! which is your friend if you want your connections to automatically reconnect if they die.

  class BaseTask
    def self.perform(*args)
    def self.perform_delegate(*args) # override this

Other than this little nit, we've found Resque to be phenomenal. Recommended.

Why the process-monitoring ruby 'god' broke

We were having all kinds of weird problems with god, and had to resort to weird hacky solutions to try to make it work. Turns out, there's a reason for it. If you try to do all sorts of shell-like actions in the commands for start/stop/restart, god ends up throwing it all in a shell process, e.g. sh -c your_cmd.

But then god gets the pid to the SHELL process and not your_cmd. And when you want to restart, it ends up killing the shell command and leaving you with zombie, orphaned, totally untracked processes that you have to kill -9 manually. Gruesome.

So in short, this stuff you should AVOID:

w.start = "cd #{RAILS_ROOT} && QUEUE=#{name} rake do:mytask"

This is OK:

w.dir = RAILS_ROOT
w.env = {'QUEUE'=> name}
w.start = "rake do:mytask"

And knowing is half the battle!

EDIT: You could also think of using Bluepill, which is God without the insane memory leak (which apparently is fixed in some hacker's dev branch!) It's in use at Serious Business, and they are one hardcore operation.

Which acts_as_solr should I use? Mattmatt's acts_as_solr, naturally.

script/plugin install git://

A note about the acts_as_solr codebase: it all started with an innocent hack that I posted to the solr-user list. It got picked up [editor 3/18/09: respectfully added a special mention of Thiago Jackiw] by Thiago and he turned it  into a serious general purpose ActiveRecord modeling plugin hosted at RubyForge, and now exists as numerous git repository forks. The currently best maintained version is Mathias Meyer’s branch.

Rails plugins have an incredibly short half-life. The creators of acts_as_solr,, seem to have disappeared. Luckily, open source can be picked up by new baton carriers.

Thanks mattmatt. You rock.

Flash security policy attacks hit Facebook/MySpace... a techie look at what happened

Cross-domain policy files (crossdomain.xml) are forgivingly parsed by Flash. If an attacker can construct an HTTP request that results in the server sending back a policy file, then Flash will accept the policy file. For instance, imagine a university website that responds to a course listing request:<cross-domain-policy><allow-access-from%20domain="*"/></cross-domain-policy>

...with the following output:

<cross-domain-policy><allow-access-from%20domain="*"/></cross-domain-policy>() {  return {name:”English101”, desc:”Read Books”}, {name:”Computers101”, desc:”play on computers”}};

Then one could load this policy via the following ActionScript? code:"<cross-domain-policy>" + "<allow-access-from%20domain=\"*\"/></cross-domain-policy>”);

This results in the Flash application having complete cross-domain access to

Github releases Resque. Wow, this is exactly what we wanted in Rails queueing. Much respect.

Resque is our Redis-backed library for creating background jobs, placing those jobs on multiple queues, and processing them later.

Background jobs can be any Ruby class or module that responds to perform. Your existing classes can easily be converted to background jobs or you can create new classes specifically to do work. Or, you can do both.

This goes into great detail into the tradeoffs and problems with existing delayed job queueing systems. Brett was mentioning -- this post reads like a laundry list of stuff we've dealt with!

Resque is a Delayed-Job-like queue that is built on Redis instead of MySQL. Brilliant.

JSON / ActiveSupport Rails gotcha: Avoid XSS exploits when passing HTML in JSON

Ran into a tricky situation today -- we're working on the ability to support javascript on Posterous blogs. One problem we saw with the Theme Editor was that </script> tags were causing problems with JSON. Browsers would see a </script> block and actually interpret as the end of the entire script block, as opposed to merely an entity within the JSON string. When dealing with user generated content, that also opens your site up to a pretty serious JS XSS attack.

ActiveSupport actually has been modified to translate < and > into their unicode encodings and avoid this problem. However, if you like many people use / require the JSON gem, this ActiveSupport to_json implementation is stripped entirely.

The simple fix -- make your own String that contains the necessary String to_json method. Be sure to use the new string class in place of the standard string class when you want the appropriate behavior of escaping angle brackets.

Here's the code:

Scaling Facebook vs Scaling Digg: It's a question of disk vs RAM

Facebook takes a Pull on Demand approach. To recreate a page or a display fragment they run the complete query. To find out if one of your friends has added a new favorite band Facebook actually queries all your friends to find what's new. They can get away with this but because of their awesome infrastructure.  

But if you've ever wondered why Facebook has a 5,000 user limit on the number of friends, this is why. At a certain point it's hard to make Pull on Demand scale.

Another approach to find out what's new is the Push on Change model. In this model when a user makes a change it is pushed out to all the relevant users and the changes (in some form) are stored with each user. So when a user want to view their updates all they need to access is their own account data. There's no need to poll all their friends for changes.

Really interesting article at High Scalability on ways to approach scaling your data store.

We use push on change as well, particularly for your reading list subscriptions. To be honest, it's cheaper. You use disk space to pre-compute things that would be expensive to ask the database repeatedly. It allows you to just add disk -- even though disk is orders of magnitudes slower than RAM.

The Facebook approach is *really hard* to get right. It's costly because so much info just has to live in RAM, and could be one reason why it's much harder for Facebook to reach profitability than most other sites. If you have to add RAM to keep all user data in cache, that's a lot of hardware to keep going.

But when it comes to realtime, Facebook is as realtime as they come. They are some real engineering badasses.

It makes no sense at all that John Resig's JS book underperforms that other turd of a JS book everyone buys

Tracking my ranking over the past year [Pro Javascript Techniques] been consistently in the 10-20,000 range, with occasional dips into the < 10,000 range. JavaScript: The Definitive Guide is always < 5,000 (for comparison).
--John Resig via

Really? That is some really really weak sauce. Pro Javascript Techniques rocks. It's a great book, and totally useful.

Javascript: The Definitive Guide, on the other hand, is a piece of turd. It routinely assumes you *already know* javascript and its prose is almost unreadable.

I wonder why Resig's awesome book is completely dominated by sales of the abysmal O'Reilly book. Probably for two reasons: a) the title (one is way more universal and likely to be bought by aspirational JS developers) and b) O'Reilly doesn't usually put out crap, so its very surprising when it does.