Identifying Customer Needs from User-Generated Content

Identifying customer needs is important to marketing strategy, product development, and marketing research. User-generated content (UGC) provides an opportunity to better identify customer needs for managerial impact. However, established methods are neither efficient nor effective for large UGC corpora because much content is non-informative and repetitive. We propose a machine-learning approach to select content for efficient review. We use a convolutional neural network to filter out noninformative content and cluster dense sentence embeddings to avoid sampling repetitive content. We further address two key questions: Are customer needs identified in UGC comparable to customer needs identified with standard methods? Do the machine-learning methods improve customer-need identification? These comparisons are enabled by a custom data set of customer needs for oral care products identified by professional analysts using industry-standard experiential interviews. The same professional analysts coded 12,000 UGC sentences to identify if each sentence contained one or more previously identified customer needs and/or new customer needs. Results: Customer needs identified from UGC are at least as valuable for product development, likely more-valuable, than those identified by conventional methods and (2) machine-learning methods improve efficiency (unique customer needs identified per unit of professional services cost).