Google lifts the lid on its once-secret server

...mounted on a motherboard built by Gigabyte. Google uses x86 processors from both AMD and Intel, Jai said, and Google uses the battery design on its network equipment, too.

Efficiency is important not just because improving it cuts power-consumption costs, but also because inefficiencies typically produce waste heat that requires yet more expense in cooling.

Costs add up
Google operates servers at a tremendous scale, and these costs add up quickly.

Jai has borne a lot of the burden himself. He was the only electrical engineer on the server design job from 2003 to 2005, he said. "I worked 14-hour days for two-and-a -half years," he said, before more employees were hired to share the work.

Google has patents on the built-in battery design, "but I think we'd be willing to license them to vendors", Hoelzle said.

Another illustration of Google's obsession with efficiency comes through power-supply design. Power supplies convert conventional AC (alternating current — what you get from a wall socket) electricity into the DC (direct current — what you get from a battery) electricity, and typical power supplies provide computers with both five-volt and 12-volt DC power. Google's designs supply only 12-volt power, with the necessary conversions taking place on the motherboard.

That adds $1 or $2 to the cost of the motherboard, but it is worth it, not just because the power supply is cheaper, but because the power supply can be run closer to its peak capacity, which means it runs much more efficiently. Google even pays attention to the greater efficiency of transmitting power over copper wires at 12 volts compared to five volts.

Google also revealed new performance results for datacentre energy efficiency measured by a standard called power-usage effectiveness. PUE, developed by a consortium called the Green Grid, measures how much power goes directly to computing compared to ancillary services such as lighting and cooling. A perfect score of one means no power goes to the extra costs; 1.5 means half the power goes to ancillary services.

Google's PUE scores are enviably low, but the company is working to lower them further. In the third quarter of 2008, Google's PUE was 1.21, but it dropped to 1.20 for the fourth quarter and to 1.19 for the first quarter of 2009 through 15 March, Malone said.

Older Google facilities generally have higher PUEs, he said; the best has a score of 1.12. When the weather gets warmer, Google notices that it is harder to keep servers cool.

Shipping containers
Most people buy computers one at a time, but Google thinks on a very different scale. Jimmy Clidaras revealed that the core of the company's datacentres are composed of standard 1AAA shipping containers packed with 1,160 servers each, with many containers in each datacentre.

Modular datacentres are not unique to Google; Sun and Rackable Systems both sell them. But Google started using them in 2005.

Google's first experiments had some rough patches, however — ,Clidaras gave one example of when they found the first crane they used was not big enough to lift a container.

Overall, Google's choices have been driven by a broad analysis on cost that encompasses software, hardware and facilities.

"Early on, there was an emphasis on the dollar per [search] query," Hoelzle said. "We were forced to focus. Revenue per query is very low."

Mainstream servers with x86 processors were the only option, he added. "Ten years ago... it was clear the only way to make [search] work as a free product was to run on relatively cheap hardware. You can't run it on a mainframe. The margins just don't work out," he said.

Operating at Google's scale has its challenges, but it also has its silver linings. For example, a given investment on research can be applied to a larger amount of infrastructure, yielding return faster, Hoelzle said.