- October 11, 2012
Pornography: Setting the Standard
- VP, Operations
If you’ve surfed the web, you’ve likely stumbled upon adult content or some reference to it. For the purpose of this article, I’d like to ignore the content shown on adult sites in favor of the content type, video, which makes these sites relevant to hosting and hosting issues. Adult content can be traced back to the early 1980s (when dial-up bulletin board systems served all the illicit content), so it’s safe to say it has been a part of the internet from the start. Neither Happy Cog nor Happy Cog Hosting work with sites that serve or publish adult content, but wherever you stand on the morality of porn, it is enlightening to consider the role it has played in shaping standards for online commerce and the way hosting providers do their jobs.
The pioneers
Internet pornography pushes hosting to the cutting edge, and it was the fuel that started other internet businesses not long ago. Google, Yahoo, and AOL all researched the porn industry to evolve their early business models. Why? Because adult content sites utilize tried and true internet business models and, as a result, are extremely profitable. In addition, sites serving adult content deal with more data than any other sites on the internet. In terms of raw bandwidth, the only sites that come close are news and media-centric sites (e.g. Hulu, YouTube, CNN), but these sites use six times less data than larger adult sites.
The hosts serving up adult content introduced many of the hosting standards we now see every day. When you make a purchase from your favorite website, you can give a nod to the internet porn industry for establishing the standards that govern credit card verification, secure payment processing, and fee-based subscriptions.
The formula
The challenges that accompany adult-centric site traffic are the stuff of hosting dreams. Building and managing an infrastructure that serves billions of pages per month and gigabytes of data transfer per second to customers is techie Shangri-la! To get a true picture of the sheer volume associated with porn traffic, I recommend Sebastian Anthony’s article, “Just How Big Are Porn Sites?.” Buried in the comments of this article is a rare glimpse into the server configuration used for one particular site. The following list reveals the hardware and software used to run a large porn site that manages 200+ million “impressions” per day:
Hardware
- 8 database servers
- 9 search servers
- 11 cache servers
- 23 app/web servers
- 17 encoding servers converting videos to H.264 for streaming and scrolling playback
- 1/2 petabyte or 512 terabytes of storage (growing at 700 GBs/day)
- 25 ad servers delivering 20 billion monthly impressions
Software
- Nginx and Lighttpd web servers
- Memcached for caching
- MySQL with Slave replication for splitting READ and WRITEs
- A thin application whose subsets reside on a different set of servers
How did the purveyors of racy images and movies arrive at this configuration (that is now largely imitated by the hosting industry)? They adopted interactive media well ahead of the rest of the internet. Static images were good, but moving images were better—so long as they moved fluidly. In order to manage the number of clicks and still maintain the speed of their sites, hosts added more hardware.
Initially, hosts added more web/app servers, but this didn’t solve the problem. As site volume increased, simply adding more web servers yielded diminishing returns. So, additional web servers were removed in favor of caching servers, and things got better. When search servers and encoding servers were added, processor-intensive functions became easier to manage. The result was increased time spent on the sites: from up to 12 minutes to up to 20 minutes. With lots of disk space and servers designated specifically for ad content, the formula for all hosting and hosting revenue was complete.
What lies ahead for hosting? If the trailblazers of the hosting industry continue as they have, we’ll see advances in anti-piracy software, artificial intelligence, and video in three dimensions. Progress in mobile streaming apps will set the technology bar even higher. In all cases, both the hosting industry, and anyone using the web, will benefit from the technologies pioneered by porn.