Why fair-proxy balancer for nginx is not a viable solution for mongrel long query queues. It's actually a disaster.

Fair-proxy balancer patch for nginx - this can keep new connections away from busy Mongrels, but if all (concurrency-disabled) Mongrels in your pool get busy, then you'll still get 502 errors.

We just hit this this morning when we tried to flip on our new and improved HAProxy + Nginx + mongrel clusters architecture. Throughput was fantastic, but I couldn't figure out why I was getting intermittent 502 errors.

Doesn't that swap one relatively annoying problem with a total fail? How is fair proxy balancer (an nginx mod) even useful in the least if when mongrels are all busy, nginx just returns a 502 Bad Gateway that completely hoses the user?

That's actually INFINITELY worse than just making someone wait 500msec longer than they should for a fast request.

I'll post again about our setup when I have all the ducks in order again, but unless something is wrong with my setup, nginx + fair proxy balancer is utterly unusable in production systems.

I'll be switching to serving nginx on the frontend for static content, with a proxy from nginx to my mongrel cluster being passed through HAProxy. HAProxy will then use the maxconn 1 flag to provide the queue-busting load balancing algorithm I wanted in the first place.

Now what to do about load balancing static content? nginx upstream reverse proxy to another cluster of nginx's? *wince*

Edit: The creator of nginx-ey-balancer contacted me and said hey, use this instead. I already switched to using haproxy for mongrel reverse proxy, but wanted to make sure to link to the github project. Looks to be a non-screwed-up version of what the fair proxy balancer was supposed to be. =) Thanks ry!

11 distributed key-value stores, as reviewed by last.fm engineers

Here is a list of projects that could potentially replace a group of relational database shards. Some of these are much more than key-value stores, and aren’t suitable for low-latency data serving, but are interesting none-the-less.

Name Language Fault-tolerance Persistence Client Protocol Data model
Project Voldemort Java partitioned, replicated, read-repair Pluggable: BerkleyDB, Mysql Java API Structured / blob / text
Ringo Erlang partitioned, replicated, immutable Custom on-disk (append only log) HTTP blob
Scalaris Erlang partitioned, replicated, paxos In-memory only Erlang, Java, HTTP blob
Kai Erlang partitioned, replicated? On-disk Dets file Memcached blob
Dynomite Erlang partitioned, replicated Pluggable: couch, dets Custom ascii, Thrift blob
MemcacheDB C replication BerkleyDB Memcached blob
ThruDB C++ Replication Pluggable: BerkleyDB, Custom, Mysql, S3 Thrift Document oriented
CouchDB Erlang Replication, partitioning? Custom on-disk HTTP, json Document oriented (json)
Cassandra Java Replication, partitioning Custom on-disk Thrift Bigtable meets Dynamo
HBase Java Replication, partitioning Custom on-disk Custom API, Thrift, Rest Bigtable
Hypertable C++ Replication, partitioning Custom on-disk Thrift, other Bigtable

Awesome review of a bunch of possible Key-value store pairs. Tokyo Tyrant conspicuously missing.