What are you doing with these lists, from a sociological standpoint?
What we are about is the thread. It turns out that the core sociological data type of the Internet is not IP (Internet Protocol) numbers, or any of that stuff, it's threaded conversations. And it's amazing how little investment has been put into adding value to the core data structure of the Internet, which is the conversational thread. I can illustrate that by suggesting that when you sit in front of your email client, simply try to sort your messages by thread size.
And by size of the thread you mean...?
I mean the number of messages, the number of generations of messages, the breadth of the conversation. If eight people reply to a message, it has a breadth of eight. If 12 reply, it's 12. And it turns out that the frequency distribution of thread properties is very illuminating.
It turns out that two-thirds of all threads in Usenet, in 2002, had a whopping two messages. And two-thirds of all authors are the people who write a message, post once one day, and never again.
Is that indicative of a spam problem?
No, those aren't spammers, they are the people who post once, get their answer and go away happy. They post a message that says they can't print, then they get their answer. What newsgroups are is a form of knowledge management application. What they are about is leveraging the collective knowledge of large numbers of people.
So how is it useful to know that people are getting their printing questions answered? What can you do with that information?
What you can do is say, "let's look at how many times each of those unique IDs posted. Twenty-four million times? That's your spammer." Humans have a limited capacity to type and send and think up messages, while software is virtually free from those constraints. What we do is say, "by looking at these properties, the structure of authors, threads and newsgroups, we can determine a lot of things that are good predictors of value."
