Smarter than Genius? Human Evaluation of Music Recommender

In this paper, we seek to gain a deeper understanding of the function and performance of the Genius recommender system by comparing it to two simple, canonical systems: one based purely on metadata, the other purely on acoustic content. We evaluate this comparison using a large user study of 185 subjects and attempt also to discover the factors that are most important for building a playlist. We discover that, in many cases, very simple metadata is adequate to produce playlists that rate as highly as or better than Genius, particularly in the short-head of familiar music that has lots of collaborative filtering data. To improve results and extend music recommendations into the long-tail, it is necessary to add content-based analysis, pointing the way to a recommender system that is smarter than Genius.