Offlining denormalized tasks - gotchas with offline queues and the after_commit callback.

In Rails, I'm not quite sure why nobody really tells you this stuff -- you kind of have to learn on your own.

We're using Workling with Starling. Things are great. (Starling is a perfectly fine queue for low-intensity tasks.) We use it for everything from sending emails to denormalizing our data.

Recently we finally fully upgraded to InnoDB for all tables. Epic win -- no more table locks! However, an unintended consequence happened. There were certain operations (e.g. adding posts to user subscriptions) that were offlined in after_save blocks. But if you look at ActiveRecord's docs, it notes:
Note that this callback is still wrapped in the transaction around save. For example, if you invoke an external indexer at this point it won‘t see the changes in the database.

Well, that's no good. Previously this wasn't an issue since MyISAM was pretty fast about commits. Turns out when using after_save, there was actually a race condition -- Workling would sometimes try to perform a job before the commit was completed, in which case we'd get "ID not found" errors since the record wasn't in there yet.

Enter the Rails after_commit plugin, which was pretty cool. It actually gives you the functionality you want if you're doing offline jobs that require the record to be complete before queueing for more work. However, our after_save callbacks were doing some optimization by checking the model.changed? flag and various related properties given to us by ActiveRecord::Dirty. But in after_commit callbacks, the model.changed? flags are reset to false, because the transaction is done.

The resolution? Move all references to the dirty bits to before_save callbacks that set flags in the instance variable. Then, when after_commit gets called, check those flags and perform the logic you want.

before_update :update_model_changed
after_commit_on_update :update_model

def update_model_changed
  @my_changed_flag = self.changed?

def update_model
  MyWorker.asynch_offline_job(:model_id => if @my_changed_flag