Wikia Search has launched. Wikia Search is Jimmy Wales’ new project, an effort to apply an “open” Wikipedia-style approach to the creation of a search engine. I haven’t been following the project closely, so I don’t know the details of how it works. But despite my reservations about the “open and transparent” Kool Aid that Wales and so many others are selling, I am glad to see such experimentation. I have been thinking a lot lately about the critical role of indexing and search infrastructure, and coming to the conclusion that there is too much research focused solely on the technological aspects of such infrastructure, and too little creative thinking about the social, economic, and political dimesions of how we provide it. Current orthodoxy seems to assume that such infrastructure should be completely provided by private companies who profit from advertising. This seems “obvious” given the success of Google, and the failure of non-commercial systems such as libraries to cope with the web. Yet I wonder if this story is too simple, and whether Google’s dominance, coupled with the radical hypercapitalist ideology that has held sway the past couple of decades, has blinded us to alternative approaches. Certainly the disadvantages of the completely privatized approach are beginning to become apparent in many areas: the troublesome co-dependence of contextual advertising and link spam, the disturbing implications of perfecting personalized search, and the temptations for private search providers to trade for their own account. Analogies between the web and the physical world are always questionable, but I wonder what the U.S. would be like if it had entrusted the construction of its transportation network, signage, maps, and such solely to private companies funded by advertisers? Would that have been the best way to support the people and companies who depend on that infrastructure to find and be found? I doubt it.

Chris Anderson has posted an absurd piece called The Probabilistic Age in which he suggests that the reason people aren’t comfortable with Wikipedia and Google is that they are systems that operate according to the laws of probabilistic statistics, which exist on some higher plane that human minds cannot comprehend. Most of the comments on the post focus on Anderson’s incoherent claim that Wikipedia somehow operates “emergently.” (This is a claim that Jimmy Wales himself disputes, by the way.) But what really concerned me was this line:

[Google] makes connections that you or I might not, because they emerge naturally from math on a scale we can’t comprehend.

There is absolutely nothing “natural” about Google’s search results. Google’s (and Yahoo’s and Microsoft’s and everyone else’s) algorithms are designed by human scientists and engineers. These scientists and engineers make specific choices about which algorithms they will use, and which they will not. They decide how the various parts of these algorithms will be weighted. They decide how they will define fuzzy concepts like “spam” and “relevance.” Each of the decisions reflects the values and preferences of the decider, and these values are reflected in the search results we see. It isn’t “alien logic,” it is human logic, and to believe otherwise is to cede control to those who write the algorithms–something I’m frankly surprised Mr. Anderson is willing to do.

When I saw Sergey Brin speak at UC Berkeley this past fall, I was very concerned when he revealed that he himself has fallen victim to, or at least wishes to propagate, the belief that his algorithms are “natural,” saying that the link structure of the web reflected the intrinsic importance of the documents linked to. But documents have no intrinsic importance–they only have importance in the context of a particular query-maker at a particular time. Sergey’s algorithms don’t reveal some truth about what is important–they encode decisions about what should be considered important. Both Mr. Brin and Mr. Anderson need to come to grip with the fact that search engines are inherently political. If people are concerned about Google, or Yahoo, or Wikipedia, then pundits like Chris Anderson should be starting discussions about what we value and how our technologies do or don’t reflect those values, not turning off their brains and blathering on about statistics and the mind of God.

The live-updating sidebar and archive of things I listen to have been updated to link to Yahoo! Audio Search, except for the few items that are in the Amazon catalog. (I feel bad ganking Amazon thumbnails without linking to them. Plus sometimes people buy music from Amazon which makes me almost eleven dollars a year.)

I’m really happy that Audio Search is finally out. This is what I was working on last summer, and it was so cool I wanted to tell everyone about it. But of course I couldn’t. I had to hold my tongue longer than expected, but I can talk about it now…. Of course my contribution to the project was extremely small, but I’m glad I had a part in it nonetheless.

Remember those bright kids I was writing about? One of them just responded to the furor over Google AutoLink by quietly releasing Butler:

Butler enhances Google search results by adding links to competitors. It also removes ads, changes typography, and a few other useful things.

Touché.

As of 8:42 this morning, the top headline on Google News was a blog. That’s a first as far as I know.

The algorithms have spoken, and the most relevant source of news on the 2004 Presidental debate isn’t a “news organization,” it’s a guy with a brain and a text editor. Looks like Dave Winer might win his bet.