Dimension Reduction of Big Data for Strategic Insights: Brand Mapping, Segmentation, and Position Dynamics from LDA vs SPCA

Consumers constantly chat online about products using a large vocabulary of words (expressing product attributes with valence or sentiments) in various online platforms such as product reviews. Each of these could scale up rapidly with variations emerging over time. The crossing of thousands of consumers’ posts by thousands of attributes and sentiments creates a large dimensionality problem of a huge, sparse matrix. Recent advances in analysis of textual data suggest techniques such as Latent Dirichlet Allocation (LDA) and Sparse Principal Component Analysis (SPCA) for dimension reduction and valence (sentiment) retrieval. First, we use LDA for extracting both the key latent dimensions and the associated sentiment (valence). Then we use extracted dimensions to gain strategic insights such as brand mapping, segment analysis, and evolution of brands perceptions over time. LDA is useful in that it can extract key dimensions and valence simultaneously. Further, thousands of attributes can be reduced to a few dimensions that bear good correspondence to results of human raters and to rankings of brands by Consumer Reports. In particular, we find that for vertically differentiated markets, objective dimensions dominate and are similar across markets. For horizontally differentiated markets, subjective dimensions dominate but vary across markets. Second, we consider SPCA for the same purpose. SPCA is a sparse statistical technique in which the meaningful variability in the data is captured by sparse principal component vectors, characterized by their corresponding eigenvalues or singular values. SPCA is more efficient than LDA but unable to extract both dimensions and valence.