Tag ruby on rails

ActiveRecord Table Transform (or, how to write to the db 27,000 times in 24 seconds instead of 9 minutes)

I implemented my new scheme and running time went from 9 minutes to 24 SECONDS. I liked this approach so much I decided to generalize it as ActiveRecord::Base.transform. Sample usage:

# if users don't have names, give them a random one
NAMES = ['Adam', 'Ethan', 'Patrick']
User.transform(:name, :conditions => 'name is null').each do |i|
  i.name = NAMES[rand * NAMES.length]
end

Really interesting use of temp tables here.

The Book of Ruby by Huw Collingbourne just released as a free PDF eBook. This looks handy.

I've largely been unsatisfied with most Ruby books (especially the oft-referenced Pickaxe book) because they don't quite go to the depth that you need day-to-day. Often a Ruby hacker needs to resort to google-fu to understand certain particularly gnarly parts of the language.

I flipped through this and it looks like a well written, well-structured overview of the Ruby language. Thanks to Huw for a great book and for providing it online here.

Offlining denormalized tasks - gotchas with offline queues and the after_commit callback.

In Rails, I'm not quite sure why nobody really tells you this stuff -- you kind of have to learn on your own.

We're using Workling with Starling. Things are great. (Starling is a perfectly fine queue for low-intensity tasks.) We use it for everything from sending emails to denormalizing our data.

Recently we finally fully upgraded to InnoDB for all tables. Epic win -- no more table locks! However, an unintended consequence happened. There were certain operations (e.g. adding posts to user subscriptions) that were offlined in after_save blocks. But if you look at ActiveRecord's docs, it notes:
Note that this callback is still wrapped in the transaction around save. For example, if you invoke an external indexer at this point it won‘t see the changes in the database.

Well, that's no good. Previously this wasn't an issue since MyISAM was pretty fast about commits. Turns out when using after_save, there was actually a race condition -- Workling would sometimes try to perform a job before the commit was completed, in which case we'd get "ID not found" errors since the record wasn't in there yet.

Enter the Rails after_commit plugin, which was pretty cool. It actually gives you the functionality you want if you're doing offline jobs that require the record to be complete before queueing for more work. However, our after_save callbacks were doing some optimization by checking the model.changed? flag and various related properties given to us by ActiveRecord::Dirty. But in after_commit callbacks, the model.changed? flags are reset to false, because the transaction is done.

The resolution? Move all references to the dirty bits to before_save callbacks that set flags in the instance variable. Then, when after_commit gets called, check those flags and perform the logic you want.


before_update :update_model_changed
after_commit_on_update :update_model

def update_model_changed
  @my_changed_flag = self.changed?
end

def update_model
  MyWorker.asynch_offline_job(:model_id => self.id) if @my_changed_flag
end

Why fair-proxy balancer for nginx is not a viable solution for mongrel long query queues. It's actually a disaster.

Fair-proxy balancer patch for nginx - this can keep new connections away from busy Mongrels, but if all (concurrency-disabled) Mongrels in your pool get busy, then you'll still get 502 errors.

We just hit this this morning when we tried to flip on our new and improved HAProxy + Nginx + mongrel clusters architecture. Throughput was fantastic, but I couldn't figure out why I was getting intermittent 502 errors.

Doesn't that swap one relatively annoying problem with a total fail? How is fair proxy balancer (an nginx mod) even useful in the least if when mongrels are all busy, nginx just returns a 502 Bad Gateway that completely hoses the user?

That's actually INFINITELY worse than just making someone wait 500msec longer than they should for a fast request.

I'll post again about our setup when I have all the ducks in order again, but unless something is wrong with my setup, nginx + fair proxy balancer is utterly unusable in production systems.

I'll be switching to serving nginx on the frontend for static content, with a proxy from nginx to my mongrel cluster being passed through HAProxy. HAProxy will then use the maxconn 1 flag to provide the queue-busting load balancing algorithm I wanted in the first place.

Now what to do about load balancing static content? nginx upstream reverse proxy to another cluster of nginx's? *wince*

Edit: The creator of nginx-ey-balancer contacted me and said hey, use this instead. I already switched to using haproxy for mongrel reverse proxy, but wanted to make sure to link to the github project. Looks to be a non-screwed-up version of what the fair proxy balancer was supposed to be. =) Thanks ry!