Microsoft used to fill its datacentres one server at a time. Then it bought them by the rack. Now it's preparing to load up servers by the shipping container.
Starting with a Chicago-area facility due to open later this year, Microsoft will use an approach in which servers arrive at the datacentre in a sealed container, already networked together and ready to go. The container itself is then hooked up to power, networking and air conditioning.
"The trucks back them in, rack them and stack them," chief software architect Ray Ozzie told ZDNet.co.uk sister site CNET News.com. The containers remain sealed, Ozzie said. Once a certain number of servers in the container have failed, it will be pulled out and sent back to the manufacturer and a new container loaded in.
It's just one method that Microsoft is using to try and cope in a world where it adds roughly 10,000 servers a month.
"You contain your infrastructure but you also contain the heat that's generated from the servers," said Arne Josefsberg, Microsoft's general manager of infrastructure, in an interview this week. "We are working incredibly hard to improve the energy efficiency of our datacentres."
Only a couple of years ago, Microsoft was adding capacity one server at a time, adding individual servers to racks and taking a couple of hours to wire in each new server.
"That's way too expensive; way too slow," said Josefsberg.
Microsoft also used to lease much of its space, until it realised that datacentres were going to be a very big part of its future as more and more software moved into the cloud.
Over the past 18 months, Microsoft has been on a buying and building spree. The company has opened a datacentre in Quincy, Washington, and will also open a Chicago facility, as well as one in San Antonio, Texas, later this year. A facility is due to open in Dublin, Ireland, next year.
Microsoft is close to announcing yet another datacentre, Josefsberg said. The software maker has also signed a memorandum of understanding to build a datacentre in Russia.
Gone are the days in which Microsoft settled for off-the-shelf hardware to fill its server farms. These days, Microsoft is looking for servers designed to its exact needs. It's not just that Microsoft doesn't want servers that have keyboard or USB ports; the company wants motherboards that don't even have the added wiring necessary to support such features, which it will never use. Such moves eliminate cost, space and power consumption.
"We are not physically building our servers, but there is very deep engagement [with the computer makers]," Josefsberg said.
Even a one or two percent reduction in power consumption makes a big difference, Josefsberg said. As it is, Microsoft is trying to cram a lot of gear into a small space. While server racks at a web-hosting facility might have power densities of 70W to 100W per square foot, the containers are packed far more tightly, potentially consuming thousands of watts of power per square foot.
The container approach is easiest to implement on the ground floor of a facility. The Chicago datacentre, for example, will use containers on the first floor and more traditional racks on the second level. However, Josefsberg said that, although the move poses some logistical challenges, the company is also considering using multiple levels of containers at other sites, including at the Dublin datacentre due to open next year.
Outsiders got a glimpse into what these datacentres are actually doing thanks to a slide included as part of a video that Microsoft put on its website touting the company's environmental efforts. The chart shows search accounts for the vast number of the servers — nearly 80,000 or so — with Hotmail and Messenger distant runners-up in terms of server usage.
Josefsberg said the figures were accurate but out of date, reflecting the situation a year or 18 months ago.
"Search was a very large portion of our demand in fiscal year 2008," he said. "Going into this year, it is still a very large proportion. It is now not as dominant as it was last year."
Microsoft is seeing new demands, he said, such as consumer video and photo services, as well as demands made by its collection of hosted enterprise services under the Microsoft Online moniker.
Josefsberg said his goal is to keep capacity a certain number of months ahead of where Microsoft's utilisation is running. To do that, he said, takes some serious planning. Business unit heads that used to have to just create a forecast for revenue and headcount, now need to be able to predict how much server capacity they will need, or at least give Josefsberg the data he needs to make such calculations.
He pointed to things like Microsoft's work with the Olympics Games as indicative of the kinds of demands his datacentres will see in years to come.
"One of the big drivers for us that I see is the move to IP-based delivery of rich video," Josefsberg said.
Not all of Microsoft's problems will be solved just because the company can now get its servers by the container. Microsoft has sophisticated 'heat maps' that plot the best locations for new datacentre based on everything from government policy to water supply to power prices. However, in other areas, such as networking technology, Microsoft is counting on the industry to make some quantum leaps.
"When you think about large-scale datacentres, there are a number of limitations in the technology," he said. "Some of the network protocols were designed years ago... some are 30 years old."
