Suggestion, Recommendation, and Filtering
As I mentioned in a previous post, Bayesian algorithms have different sets of criteria depending on the expectations of their users. I find three levels that applications fall into.
Purely implicit suggestion algorithms by definition have the lowest level of interaction - they measure what other users have been interested in in the past, compare it to the things you do, and using some reasonably complicated math make suggestions based on that information. Targeted advertising campaigns make sense for this approach - most people aren't willing to make an effort to make it easy to advertise to them, and expectations are zero that you'll be advertised to, and you don't need any trust at all to make them effective.
Amazon is an example of a purely implicit approach taken as far as it can go... and maybe a little too far. An advertising algorithm that directly interacts with users by making explicit recommendations - it tells me "Hey Adam, we think you'd like this..." as opposed to a passive advertising engine like adsense. Unfortunately since the algorithm is implicit I don't have a way to tell the algorithm that the South American Flute music CD was a gift 5 years ago, and I have no interest in the emails they send me suggesting other world music.
Without any more information than the things I buy or view however, the ability for this algorithm to provide useful suggestions faces an asymptote as the math behind it gets better and better - my interest profile snapshot at this moment in time is much more complex than can be gleaned from my historical shopping data.
Recommendation is the next tier up from suggestion. The difference is that suggestion supplies data without a question being asked, while recommendation is in response - potentially abstract - to a question which the user has posed. Pandora, Digg, Delicious - we have become accustomed to interacting with applications that take our input and use it to make direct recommendations. They still use Bayesian algorithms of course, but they are fuelled with direct feedback, which can eventually provide enough information to make pretty good recommendations about things you might be interest in.
True filtering is a lot more serious. Filtering is as far from recommendation as recommendation is from suggestion, because it requires trust. You provide the raw data you want filtered, and then trust the system to give you the subset you are actually interested in and not cut out the things you consider critical. Spam filters are a great example of this. When my radio station plays something I don't like, it is much less serious than when the filter deletes that email from grandma. Alex Iskold wrote an interesting article that was applicable for netflix, but it's not a good fit for filtering as we're talking about it - when filtering messes up the analogy isn't like finding a restaurant that wasn't recommended but you liked anyway; it's more like finding out all your friends had a party that they didn't invite you to because they didn't think you'd be interested in.
I don't want to talk too much about the stuff we're working on, but I can say this: if you're primarily focused on the wisdom of crowds, ur doing it wrong. Your holy grail is the Greatest Newspaper In The World, which... misses the point. There are a lot of editors at every major paper in the world who get up every morning and work their butts off to do just that... and then Google News and Reuters skim off the cream. If you want to cobble together the world's best most popular newspaper, you're playing for scraps. No, the goal of the filtering web 3.0 evolution is going to be to finish the job RSS started, to create your ideal newspaper. Stay tuned.
Purely implicit suggestion algorithms by definition have the lowest level of interaction - they measure what other users have been interested in in the past, compare it to the things you do, and using some reasonably complicated math make suggestions based on that information. Targeted advertising campaigns make sense for this approach - most people aren't willing to make an effort to make it easy to advertise to them, and expectations are zero that you'll be advertised to, and you don't need any trust at all to make them effective.
Amazon is an example of a purely implicit approach taken as far as it can go... and maybe a little too far. An advertising algorithm that directly interacts with users by making explicit recommendations - it tells me "Hey Adam, we think you'd like this..." as opposed to a passive advertising engine like adsense. Unfortunately since the algorithm is implicit I don't have a way to tell the algorithm that the South American Flute music CD was a gift 5 years ago, and I have no interest in the emails they send me suggesting other world music.
Without any more information than the things I buy or view however, the ability for this algorithm to provide useful suggestions faces an asymptote as the math behind it gets better and better - my interest profile snapshot at this moment in time is much more complex than can be gleaned from my historical shopping data.
Recommendation is the next tier up from suggestion. The difference is that suggestion supplies data without a question being asked, while recommendation is in response - potentially abstract - to a question which the user has posed. Pandora, Digg, Delicious - we have become accustomed to interacting with applications that take our input and use it to make direct recommendations. They still use Bayesian algorithms of course, but they are fuelled with direct feedback, which can eventually provide enough information to make pretty good recommendations about things you might be interest in.
True filtering is a lot more serious. Filtering is as far from recommendation as recommendation is from suggestion, because it requires trust. You provide the raw data you want filtered, and then trust the system to give you the subset you are actually interested in and not cut out the things you consider critical. Spam filters are a great example of this. When my radio station plays something I don't like, it is much less serious than when the filter deletes that email from grandma. Alex Iskold wrote an interesting article that was applicable for netflix, but it's not a good fit for filtering as we're talking about it - when filtering messes up the analogy isn't like finding a restaurant that wasn't recommended but you liked anyway; it's more like finding out all your friends had a party that they didn't invite you to because they didn't think you'd be interested in.
I don't want to talk too much about the stuff we're working on, but I can say this: if you're primarily focused on the wisdom of crowds, ur doing it wrong. Your holy grail is the Greatest Newspaper In The World, which... misses the point. There are a lot of editors at every major paper in the world who get up every morning and work their butts off to do just that... and then Google News and Reuters skim off the cream. If you want to cobble together the world's best most popular newspaper, you're playing for scraps. No, the goal of the filtering web 3.0 evolution is going to be to finish the job RSS started, to create your ideal newspaper. Stay tuned.
posted by aseever at
10:13 PM
0 Comments:
Post a Comment
<< Home