Work(ing Title)

When it came time to ask someone to refine the idea behind the Defrag conference, Jeremy Zawodny (Yahoo) was tapped to provide an explanation. The first ideas Jeremy felt were the reasons for Defrag to exist were Open Source, Social Everywhere, Web Services, and Web platforms. Open Source got left out of the discussion a bit, but the remaining three were very interesting.

The point was made the MySpace and Facebook typify sites that are built as destination sites without much real function. In contrast, MyBlogLog attempts to incorporate social software around online activities people already participate in.

This argument makes plenty of sense up front. Providing rich socialization ability in the online locales where users already spend a lot of time is both obvious and logical. Personally I get all dizzy at the thought of a Facebook-esque news feed built into my Flickr homepage. However the question of a tipping point is worth exploring; If enough users all show up, does a site need a real purpose? There are plenty of popular bars that have lines out the door and tonnes of people inside, while the only function of the place is to buy overpriced drinks.

Yahoo’s investment in Hadoop, in conjunction with the in-house growth of Yahoo Research’s Pig, represent a shift in development priorities towards something Jeremy calls, “infrastructure software.” The social networks, even in their first-generation ‘outpost’ modes, are showing us that software needs to leverage grid/infrastructure computing, as distribution is handled by the users.

The movement towards developing software to run in the cloud is clearly a mandate for anyone who wants to have an audience moving forward. When you combine the idea of relocating social software and building the tools for developing infrastructure software, the end result is an extremely democratized method of software deployment. The human experience of finding software through your social grid is more capable than traditional marketing and distribution. This is incredibly similar to the painful evolution of the music industry, and I think what application developers need is something like a Radiohead vanguard to follow.

This morning’s Defrag keynote opened with a panel discussion on where the business of search and discovery has come from, and where it needs to go.

Marti Hearst (UC Berkeley) made a strong opening analogy by explaining that search is currently an, “experience in orienteering;” the journey to your information begins with a few words, followed by a long period of sniffing out clues. This process repeats until hopefully you find what you were looking for.

Obviously this creates a demand for a major upgrade to the human experience of search. One of the natural solutions is natural language processing, where Microsoft is doing a lot of great work (TellMe, Windows Speech Recognition, and Ford Sync).

Another of the interesting factors in the evolution of search is a pitch for building more topic-specific indexes. Steve Larsen (Krugle) argued a compelling case for vertical search indexes, “In a code-writing search index, Python is never a reptile and always a language.”

This feels like a workaround for the lack of NLP implementation/effectiveness, but as opposed to most instances, that sort of design sounds extremely valuable. I think I might rather have a half-dozen different search engines that I use across a day, assuming they are each much more targeted toward the information I’m looking for. I think this type of strategy could allow for a lot of smaller-scale growth in the search market.

Bradley Horowitz (Yahoo) gave an explanation of Flickr’s Interestingness system which brought up the powerful difference between explicit and implicit discovery systems. Explicit systems (voting, rating) are ripe for gaming and rigging, which obviously prevents an honest view of the landscape from emerging. The Interestingness recipe watches views, comments, and favourites across all the system’s photos. In addition though, it weights the value of those actions based on the relationship the author has with the viewer (your brother marking your photo as a favourite is different than a stranger doing the same). Intricacies like that help develop a more honest result set.

The beauty I see in these implicit designs is their invisibility to a user’s actions. Rather than putting the request on the user to think consciously about the value they place on something, the software simply listens and reacts. As Bradley put it so eloquently, “The system changes in the user’s wake.”

When the discussion got to addressing the uncrawlable information trapped behind paywalls and corporate firewalls. Jeremie Miller (Search Wikia) made the comment that as people define knowledge by what is discovered through search, knowledge that doesn’t make it into the index may cease to exist (so far as many people are concerned).

Honestly I’m terrified by the social implications of such a reality; the rift between those with access to the indexes and those without can have dire effects, not to mention the concerns around who controls those indexes.