Web ratings: Numbers can lie

NEWS
Jerry Ziegler thought he'd hit the jackpot last August when his company, eFront Media, was listed as one of the 20 most-trafficked sites on the Web. "There was euphoria at the office," the former eFront president recalled of the July ranking by research firm Media Metrix. "We were having trouble raising the institutional money, but once those numbers came out, we got a commitment for a very large amount of money. It was a serious deal, with cash and connections, and it fell right into our laps because of Media Metrix." There was just one problem: the ranking was a mistake. The Web site aggregator's leap to the big leagues was based on incorrect data. When Media Metrix learned of the error several weeks later, it revised the monthly rankings -- the only time the Net research company has ever had to restate its top-50 list. After the revision, eFront didn't even make the top 500 for July, although in later months it managed to crack the top 50. The eFront case raises concerns about the soundness of the influential system that has become the de facto standard for counting Web traffic -- important figures followed closely by companies, advertisers, investors and industry analysts. Although its methodology has long been considered inexact, Media Metrix rose quickly to prominence in the heady days of the Internet economy when traffic was the equivalent of digital gold paving the way toward initial public offerings. Among the questions to emerge from the eFront controversy: How was the incorrect information submitted? Why wasn't it discovered before the rankings were issued? Have other Media Metrix rankings been tainted in similar cases that have not come to light? "The whole system is set up based on some trust that the people they are measuring are not just making things up out of whole cloth," said John Corcoran, an analyst at CIBC World Markets. Although traffic numbers provide "valuable information," he added, "this incident with eFront... is not helpful to Media Metrix." Media Metrix is hardly the first media measurement company whose numbers have been challenged. The Nielsen ratings, which extrapolate TV viewership figures from a handful of families surveyed, have been criticised for years. (Nielsen, in conjunction with online research company NetRatings, also measures Web traffic in competition with Media Metrix, which merged last year with market researcher Jupiter Communications to become Jupiter Media Metrix.) Despite their detractors, Nielsen and Media Metrix have become widely accepted in the Internet and TV industries, respectively, as no alternative systems have proven more accurate. Unlike the TV ratings, however, online rankings can have far more urgent consequences: Traffic can be the fine line between a company's life and death in the fragile Internet economy, with investors not far behind. Still, Media Metrix's revision of eFront's numbers drew little public attention until last month, when hundreds of pages of ICQ instant messaging logs between eFront chief executive Sam Jain and his correspondents, including some fellow executives, were apparently pilfered from his PC and posted on the Web. The logs, which read like transcripts of telephone conversations, include explosive and candid discussions regarding business partners, employees and affiliated Web sites. But among the most damaging allegations to arise after the publication of the logs is that eFront intentionally misled Media Metrix to get the company listed among the Net elite. Although the rankings episode is not detailed in the logs, it has been the subject of many message-board postings and has been acknowledged by Media Metrix, eFront executives and affiliates of the company. "It is what it is -- it's obviously not a positive thing," said Stacie Leone, a Media Metrix spokeswoman, who described the incident as an anomaly. She said the company uses a 20-person team to keep track of and verify "roll-ups," or lists of Web addresses assigned to particular properties, on a monthly basis. "Because there are hundreds of those URLs coming at us... they need to provide us with documentation that they outright own or have a partnership with the company that has agreed to let them roll up that property into their traffic," Leone added. As the eFront fiasco illustrates, however, this trust-but-verify approach is not foolproof. In an October press release, Media Metrix said it revised the rankings after learning that eFront "unintentionally submitted inaccurate information". But Ziegler and two other eFront executives told CNET News.com that Jain was responsible for submitting data he knew to be inaccurate. Jain denied the charge and blamed Ziegler for filing the paperwork with Media Metrix. Regardless of who was responsible for the submission, the episode underscores the potential for abuse of the counting system used by Media Metrix. Although the company maintains that it would have caught the discrepancy eventually on its own, it acknowledges that it discovered the problem after being alerted by a rival of eFront. "In an ideal world, yes, these URLs are checked before the numbers are issued. But many of those sites are small mom-and-pop sites, and it's a matter of checking hundreds and hundreds of URLs, which would hold up the entire reporting process," Leone said. "In the case where they have so many partners, we might also do a basic spot check [of the URLs] and get most of the letters [from both parties verifying who should get credit for the traffic]. I can't say for certain whether we did or not in the eFront case." The research company defends its system and has not changed any procedures following the eFront controversy. "Media Metrix felt we had sufficient evidence to aggregate this site and report these numbers. Some of the information turned out not to be right," Leone said. She added that Media Metrix executives consider the incident "a blip on their radar." When it changed eFront's ranking, Media Metrix reclassified the Web site's designation from a "property" -- a branded entity that encompasses other sites, such as the way the Microsoft Network includes Expedia and CarPoint -- to a "domain". As a domain, eFront was no longer able to count traffic of affiliated sites, relying instead only on traffic generated at its primary site. Yet the revision largely escaped major news media. Even after the eFront ICQ scandal, a Los Angeles Times story described the company as being "among the 20 most-frequently visited Web sites, garnering about 12.4 million viewers in July, according to Media Metrix". There is no mention of the restated rankings in the March 17 story. The intoxicating power of such high traffic is precisely what drew Jennifer Moss, Webmaster for BabyNames.com, to eFront. The company's concept was to sell advertising across a network of hundreds of small Web sites, many it had bought in exchange for eFront equity and the promise of three years' monthly payments. But Moss and other Webmasters say they received at most a handful of the payments and are now seeking to regain control of their sites. Ziegler concurred that eFront had trouble making payments to its affiliates. Throughout 2000, eFront was rapidly acquiring an eclectic mix of Web sites with the goal of taking the network public. The efforts appeared to pay off with the July top-50 list, which caught the attention of Salon.com. Last August, the online magazine wrote that the buy-and-hold strategy drove eFront "to number 18 on the Media Metrix list, with 12.5 million visitors. The 14-month-old company is a huge roll-up of some 150 small content sites, ranging from gaming to tech news and loopy entertainment." EFront trumpeted the BabyNames alliance in a press release dated 3 October -- one day after Media Metrix issued its revised top-50 list and bumped eFront off it. Rather than having 12 million unique visitors, eFront had fewer than 400,000 "uniques", according to the revision. That was particularly bitter news to Moss, who said she had agreed to sell her Web site to eFront and become an affiliate partly because of its apparent success in cracking the top 20. "The month after we signed with them, they were suddenly [removed from] the charts," lamented Moss, who also is owner of Los Angeles Web consultancy Moss Mallory Interactive. "That bothered us because that particular ranking was one of the major reasons that made our decision to go into this deal." Former eFront chief technology officer Matt Levine, who resigned after the ICQ logs became public, called the Media Metrix episode the beginning of the end for several of eFront's executives. "Media Metrix was the straw that started putting pressure on the camel's back," Levine said. "That really started to put strains on relationships." That has become abundantly clear in the dispute between Jain and former company President Ziegler, who has filed a lawsuit in Superior Court in Orange County, Caliornia, against eFront and Jain seeking back pay. "Sam [Jain] would call companies all over the world, trying to get them to be our affiliates," Ziegler said. "Some of them wouldn't respond. He said, 'If they won't respond to me, they won't respond to anyone.' Sam took these and submitted them to Media Metrix and said they were affiliates." Jain called the allegations "absolutely not true," saying in an interview that "Jerry [Ziegler] himself was in charge of our affiliates program. I had nothing to do with it." Moss doesn't particularly care how Media Metrix got the bad information or why it wasn't discovered sooner. At this point, she has more immediate concerns. Like other Webmasters who did business with eFront, Moss says she has not received the promised monthly payments. She wants to regain control of the BabyNames Web site, arguing that the revised Media Metrix numbers provide a legitimate reason to break the affiliation because traffic was a major motive for joining eFront. "We can dispute whether the original contract was brought about in good faith," Moss said. "We believe it wasn't." CIBC analyst Corcoran said eFront was hardly the only Internet company apparently obsessed with traffic numbers. It was just a "more visible example that caused a little more embarrassment", he said. To understand companies such as eFront, Corcoran said, it is important to view them in the broader context of the Internet gold rush -- and the lengths to which some people may go to strike it rich. "There was a big temptation to pump up the numbers two years ago," Corcoran said. "These companies had business models where they knew they weren't going to show any profit for some time, so companies were judged by an unusual set of criteria: page views and eyeballs." Take me to ZDNet Enterprise Have your say instantly, and see what others have said. Click on the TalkBack button and go to the ZDNet News forum. Let the editors know what you think in the Mailroom. And read what others have said.

Post your comment

In order to post a comment you need to be registered and logged in.

You can also log in with Facebook. Log in or create your ZDNet UK account below

  • Login

Will not be displayed with your comment

By signing up for this service, you indicate that you agree to our Terms and Conditions and have read and understood our Privacy Policy. Questions about membership? Find the answers in the Community FAQ

Get ZDNet UK's daily newsletter

Enter your email address to sign up

ZDNet UK Live

BrownieBoy

@Jack, > Works really well for thieves.... Nice attempt to deflect the argument by tossing in a point that's totally irrelevant, even it were...

6 hours ago by BrownieBoy on AMD Ultrathins to challenge Intel Ultrabooks
bootlegger

Make that 13 people now - I got refused today at Manchester airport. I thought I was up to date on this legislation - I knew of the EU ruling from...

9 hours ago by bootlegger on UK airport body scans will not be opt out
tinycg

Don't forget to check out apps like GoodReader or SlideShark either, they're indispensible for people on the go in presentation situations. Best...

12 hours ago by tinycg on Four top iPad apps for people on the move
TerryRK

Well it seems there is something a number of us agree on. Why is the Ubuntu Unity launcher so ugly? I thought perhaps it was something to do with...

16 hours ago by TerryRK on A tale of two distros: Ubuntu and Linux Mint
Freebies202

Duplicate comments are not made intentionally. Its very good to know that now you are keeping check on this problem because sometimes a commenter...

1 day ago by Freebies202 on Microsoft fixes blog comments, speeds up blogs with open source
kevinmchapman

"the very significant number of users" and "many (most) of us" - you have no evidence for these statements. It is a fact that most users are saying...

1 day ago by kevinmchapman on A tale of two distros: Ubuntu and Linux Mint
Marg Menzies Harrison

Another grammar faux pas is the improper use of "you". When sitting down down in a restaurant, for example, I get cringe when the waitress...

1 day ago by Marg Menzies Harrison via Facebook on 10 flagrant grammar mistakes that make you look stupid
zdnetukuser

And NOW, folks, for Canonical's next trick... Kubuntu is late. Here's a pencil. Draw your own conclusions. cf.:...

1 day ago by zdnetukuser on Linux Minterface
Moley

@kevinmchapman. The discussion here reflects the very significant number of users who really do like the traditional menu system and who wish to...

2 days ago by Moley on A tale of two distros: Ubuntu and Linux Mint
kevinmchapman

Er, no... It is an efficient means of finding the application/file/setting you need in one place. The icons are a simply a fallback for when you...

2 days ago by kevinmchapman on A tale of two distros: Ubuntu and Linux Mint
TerryRK

Isn't the provision of a text based search an admission by the developers that the mass of icons approach does not work? I don't need to use a...

2 days ago by TerryRK on A tale of two distros: Ubuntu and Linux Mint
kevinmchapman

"Unity and GNOME 3 both abandon the old text-based cascading menus in favour of a graphical icon-driven system." Point truly missed. Both use a...

2 days ago by kevinmchapman on A tale of two distros: Ubuntu and Linux Mint
TerryRK

whs001 - Thank you, I'm glad you liked the article. I absolutely agree with you on your first point. I should perhaps have made it clearer that...

2 days ago by TerryRK on A tale of two distros: Ubuntu and Linux Mint
Dennis Nilsson

If we allow corporate interest to dictate the way our government circumvents due process against foreign entities then we should accept the same...

2 days ago by Dennis Nilsson via Facebook on ACTA stumbles in Germany
GHar123

I totally dislike pirating of works, I fear that artists will be deterred from creating works if they think that they are going to get ripped off....

2 days ago by GHar123 on ACTA stumbles in Germany
JCB33

How dare film makers, artists or anybody that invests in creativity stop us pirating their works for free. I want to be able to walk into my local...

2 days ago by JCB33 on ACTA stumbles in Germany
Moley

@GrueMaster. I prefer horses for courses rather than one size fits all. I, and I suspect most other computer users, do not really wish to have...

2 days ago by Moley on A tale of two distros: Ubuntu and Linux Mint
greycynic

The product that scares me every time I have to use it is the Office 2007 version of Excel. The first bug that I found was applying the median...

2 days ago by greycynic on Ten flawed products that derail productivity
GrueMaster

Nice review and very informative. One thing I'd like to add (in reply to whs001's 1st question), the main reason to have the same interface from...

2 days ago by GrueMaster on A tale of two distros: Ubuntu and Linux Mint
Frederick Wrigley

I'be been using Mint 12 since the RC came out, and I am far more happy with the Cinnamon, the Mate, and, yes (with extensions), theGnome 3...

2 days ago by Frederick Wrigley via Facebook on A tale of two distros: Ubuntu and Linux Mint