DomainSponsor is Oversee’s domain parking division. When people have domain names that they’re not ready to use yet, we show ads on them. DomainSponsor gets around a billion visits per month.
We needed an efficient, but extensible, protocol for these servers to communicate with each other. At previous jobs I’d worked on other projects where we rolled our own binary protocol for similar distributed systems, and I’d seen the headaches involved in upgrading the protocol version, even just to change the format of one field. We looked around, and found several fairly similar libraries for extensible binary protocols. The main open source competitors at that time (mid-2008) were Protocol Buffers and Thrift. Protocol Buffers (or “protobufs”) had just been released by Google, after several years of internal use. Thrift had been created by some ex-Google interns working at Facebook, and so the two are fairly similar. Both produced very compact on-the-wire encodings, though protobufs, with its variable-length numeric fields, had a slight edge. Both were designed to allow easy extensibility; new optional fields could be added without breaking existing code (which would just ignore the new fields).
We chose protobufs for several additional reasons, not that overwhelming individually, but combined they made protobufs a better choice for us. First, the documentation of protobufs seemed more complete and thorough. Second, their Perl support, while not part of the core project, was better suited to our plans to write non-blocking event-driven clients and servers. Thrift’s Perl support was very complete on the client side, but there was no server-side implementation, and the client code did its own blocking network I/O, which would be hard to integrate with an event-driven framework. There are actually at least 3 Perl bindings for protobufs. protobuf-perl was the first pure-Perl implementation. We initially tried protobuf-perl, but it had some serious performance issues, partly due to teething issues with Moose, which was still relatively new and unoptimized at that point. At that point, protobuf-perlxs was released, which puts a Perl wrapper around the C++ code generated by Google’s stock protocol buffers compiler. It has the disadvantage that it requires compiling and linking to C++ code, which makes it slightly harder to port and install on unusual architectures, but its speed advantage over pure-Perl implementations more than makes up for that. Google::ProtocolBuffers is a newer pure-Perl implementation that includes its own parser for the .proto interface description file format, and is the only implementation in CPAN; it was released after we’d finished our evaluation of the other two implementations. It supports creating message definitions at run time (as opposed to build time), which could provide some interesting flexibility, but for our purposes protobuf-perlxs’s speed is more important.
When it comes to just about any need, the Comprehensive Perl Archive Network (CPAN) provides an embarrassment of riches. There are several popular event-driven frameworks for Perl. The largest and most mature is POE; a more recent competitor is AnyEvent. The main author of POE, Rocco Caputo, has been working on a cleaner re-implementation of POE using Moose, called Reflex. There are also some smaller event-driven libraries that aren’t as all-encompassing: Event::Lib, Event, EV, and IO::Async. For our purposes, at the time we made the decision, POE provided the most useful components for things we needed, such as asynchronous DNS queries, fully-featured HTTP request processing (including keepalive, HTTPS, etc.), and the ability to easily support our own protocol.
Today, as a result, we have a large distributed system handling a huge number of requests across an easily-scalable cluster of machines, it’s relatively easy to write new servers using our POE-based framework. We can also add new fields to the messages we send between the tiers, without having to worry about compatibility issues when old and new versions of the code talk to each other.