Krisztian posted a link to the TREC 2009 Entity Track Overview, part of the TREC 2009 proceedings.

The track website has information on the 2009 track and what is planned for 2010. One change they are seeking discussion about is a new semantic entity search subtask:

We propose a semantic entity search subtask for 2010: return URIs of related entities, instead of their homepages. We are planning to enrich topics with URIs of the input entities. URIs need to come from a predefined set of semantic data sources (which will include DBPedia and Freebase, at least).

The plan is to use the full category A set of ClueWeb09, which has 500 M English web pages instead of the smaller B subset which doesn't contain many entity homepages.

An updated draft of the upcoming book, Data-Intensive Text Processing with MapReduce by Jimmy Lin and Chris Dyer is available.

The book isn't finished, but it still has interesting material. It emphasizes algorithms for processing text with Mapreduce: co-occurrence analysis, inverted index construction, and the EM algorithm applied to estimating parameters in HMMs.

Peter Mika highlights the Semantic Search competition at the upcoming Semantic Search 2010 workshop at WWW 2010. From Peter's post,

Participants will be given queries sampled from a web search query log provided by the Yahoo Webscope program, and have to try to answer those queries using the Billion Triples Challenge corpus from 2009. The queries that are selected are all entity queries in that they are looking to find information about a single entity.

This is an interesting competition because it attempts to use unstructured web queries to do retrieval over a heterogeneous collection of structured data. The Billion Triples collection contains data from DBpedia (extracted from Wikipedia), Geonames, a variety of social networks, and other sources.

Yahoo! has announced a Learning to Rank challenge as part of the Learning to Rank Workshop at ICML 2010.

They are releasing (to participants) two large real-world datasets. The first dataset has:
29,921 queries
744,692 URLs
519 features

For details on the second set, see the website.

The URLs are rated on a graded scale, 0 (irrelevant) to 5 (perfect). The evaluation will use Normalized Discounted Cumulative Gain (NDCG) and Expected Reciprocal Rank (ERR).

The set only includes query and URL identifiers without the original information, so engineering new features seems unlikely.

The competition begins March 1st and goes through May 31st.

I haven't been writing much recently. I was a bit burnt out after paper season. I submitted a short paper on synonym recognition to ACL 2010. I hope to share more on that in the future. On the topic of synonyms, the recent Wired article on How Google's Algorithm Rules the Web mentions briefly their synonym recognition algorithm.

Towards the middle of the article, Amit Singhal talks about synonyms. The first part talks about the straightforward mappings identified from query reformulations. I think the more interesting case is when you don't have millions of those to learn from. You can use the information on the web documents. Here's the relevant section,

Google’s synonym system understood that a dog was similar to a puppy and that boiling water was hot. But it also concluded that a hot dog was the same as a boiling puppy. The problem was fixed in late 2002 by a breakthrough based on philosopher Ludwig Wittgenstein’s theories about how words are defined by context... “Today, if you type ‘Gandhi bio,’ we know that bio means biography,” Singhal says. “And if you type ‘bio warfare,’ it means biological.”

This type of query sensitive synonym usage is quite important for web retrieval.

See also my recent previous post on Google's synonym effectiveness and their recent patent on using query context for determining synonyms.

Today is the beginning of the Web Search and Data Mining conference, WSDM 2010. You can follow the conference on Twitter, #wsdm2010 as Daniel Tunkelang, Ian Soboroff, and others are tweeting it. The proceedings are also available. I hope to highlight some of the papers here in future posts.

Yesterday was the Search in Social Media (SSM) 2010 workshop. Daniel posted his summary. You can also read the tweet stream.I look forward to reading more coverage and getting other perspectives on the event.

Today is the Search in Social Media Workshop, SSM 2010. Unfortunately, I'm not attending, but you should follow the coverage on Twitter, #SSM2010. You can read the papers from the program. However, the really interesting parts will be the Keynote from Jan Pederson and thee panel discussions:I look forward to reading the coverage!

I'm starting a new project. We're building a prototype search system, so we want it to be quick and simple. This started me thinking about options...

SearchEngineLand has an interview with Stefan Weitz, a Director at Bing Search. The interview highlights some of Bing's future directions and challenges in search. One emphasis is on better supporting user's tasks. Stefan says,

Relevancy is relative. It is about the intent of the user, first of all. What is the user trying to do? Then, secondly, what do you know about the user or the query that could help to better refine the results?

I disagree with the above statement. By definition, a document is relevant if it satisfies a user's "information need". I think he's really trying to say that too often we make the mistake of removing the user from the equation and creating a universal relevance judgment that holds across all users who issue a query.

The interview goes on to talk about how Microsoft is investing in technologies to support complex decision making, in vertical categories like travel, health, and shopping. He highlights Farecast. However, it's also clear by the current Bing results for plan a trip to Florida, that there is still a long way to go.

The article goes on to detail Microsoft's "vertical" strategy as a means of differentiation:

We will continue to introduce these verticals, in pretty short order, frankly. The sum of those parts will become a very differentiated experience that will expand how people think about search...

If you missed it, yesterday Microsoft started rolling out Bing Recipe Search.

Google has used synonyms for query expansion for several years now. It is part of their attempt to find what you mean, not just what you type. Steven Baker, an engineering on the quality team wrote a post covering a recent examination of synonym usage in query expansion. He writes,

...our measurements show that synonyms affect 70 percent of user searches across the more than 100 languages Google supports. We took a set of these queries and analyzed how precise the synonyms were, and were happy with the results: For every 50 queries where synonyms significantly improved the search results, we had only one truly bad synonym.

Another tidbit is that Google is expanding their highlighting of synonyms in search result summaries.

Lastly, a tip if you get stuck with one the 1 in 50 queries where synonyms go bad:

You can also turn off a synonym for a specific term by adding a "+" before it or by putting the words in quotation marks.

Bill Slawski has good coverage of the post, and previous work on synonym usage, including Steven's patent, Determining query term synonyms within query context.

I'm a bit behind here because of the upcoming SIGIR deadline. However, I wanted to make sure to mention an article in the January CACM, New Search Challenges and Opportunities. The article highlights three main directions:

Yesterday, Technology Review posted an interview with Amit Singhal, on How Google Ranks Tweets. According to Amit, one key is to find "reputed followers",

Jeff Dean and Sanjay Ghemawat wrote an article for the January edition of CACM, MapReduce: A Flexible Data Processing Tool. In the article, they refute the findings of A Comparison of Approaches to Large-Scale Data Analysis. On their blog, the authors also wrote a post bashing MapReduce: MapReduce, A major step backwards. The post is no longer available, but thankfully Greg had good coverage.

In the article Dean and Ghemawat address the paper and attempt debunk its claims, although they lack the benchmarks to back it up. In the process, they inform you about the right way to run M/R jobs efficiently:

1. Avoid starting processes for each new job, reuse workers.
2. Careful data shuffling, avoid O(M*R) disk seeks
3. Beware of text storage formats.
4. Use natural indices like timestamps on files.
5. Do not merge reducer output.

They present some good M/R lessons in their refutation. You should be using a binary serialization system like Avro or Protocol Buffers and storing your data in a format that provides efficient access, using a natural file structure or using a database system like HBase.

via Dmitriy Ryaboy on Twitter.

The NY Times ran a op-ed article, Search, but you may not find. I can't believe they ran such rubbish. I'm not going to bother to debunk it, Paul Kedrosky did a better job than I could.

I'm writing a Hadoop job and I ran into a little problem that I wanted to share (and remind myself of the solution for the future).

I am packaging up my Hadoop program into a Jar file. It has external dependencies on text parsers. To include these with my program, one way to do this is to package the dependencies inside the jar in a /lib directory. This ensures the jar and all dependencies get copied to the Hadoop Mappers.

I create my jar file by right-clicking on the project --> export --> Java --> Jar file. I then select my code and the lib directory. However, the problem I had was that my lib directory was not being exported. I learned that this happens if the jars in lib are on your build path. To solve this, the jars need to be "external" or in a different folder. Then you can export the lib directory as a resource.

Anyone care to share a better solution?

Today, Google announced an extension, Quick Scroll, for Chrome 4 Beta that utilizes your previous search while browsing. Consider their example query [belgian waffles served by street vendors?] and browsing a result:

... a small black box appears in the lower right hand corner of the browser with a couple snippets of text from the page that might be relevant to your query.... Quick Scroll analyzes things like proximity, prominence and position of the words to identify the most relevant content.

... who will be the first to reverse engineer the formula from the extension?

The ICWSM is a conference on blogs and social media. For the conference, they issued a data challenge.

The dataset, provided by Spinn3r.com, is a set of 44 million blog posts made between August 1st and October 1st, 2008. The post includes the text as syndicated, as well as metadata such as the blog's homepage, timestamps, etc. The data is formatted in XML and is further arranged into tiers approximating to some degree search engine ranking. The total size of the dataset is 142 GB uncompressed, (27 GB compressed).

The deadline is March 1st.

Something to look at after the SIGIR deadline....

Google is hosting a press event at the Computer History Museum offering an "inside look at the evolution of Google search".

Danny Sullivan is live-blogging the event at SELand.