Dynamic Text-Based Industries and Endogenous Product Differentiation

We study how firms differ from their competitors using new dynamic measures of product differentiation based on novel text-based analysis of 50,673 product descriptions from firm 10-K statements filed yearly with the Securities Exchange Commission. This year-by-year set of product differentiation measures allows us to generate a set of dynamic industries and new measures of industry structure and competition. These competitiveness and market structure measures better correlate with firm profitability than do existing classifications. Using these new dynamic industries, we examine endogenous product differentiation. We show that firms use R&D and advertising to differentiate themselves from competitors and increase profitability. ∗University of Maryland and University of Maryland and National Bureau of Economic Research respectively. Hoberg can be reached at [email protected] and Phillips can be reached at [email protected]. We especially thank Dan Kovenock, Steve Martin, Rob Porter, John Sutton and seminar participants at HEC, IFN (Stockholm), Insead, ISTCE (Lisbon), London Business School, Northwestern University, Notre Dame, Stockholm School of Economics, University of Amsterdam and University of Vienna for helpful comments. All errors are the authors alone. Copyright c ©2009 by Gerard Hoberg and Gordon Phillips. All rights reserved. Defining industry boundaries and industry competitiveness is central to the study of industrial organization. It is also central to broader disciplines in Economics and Finance, where the study of industries, or the need to control for industry, is pervasive. Our paper is based on the premise that product similarity is core to classifying industries, and that empirical work can benefit from the ability to measure industry memberships and product differentiation in every year. Using new dynamic industry classifications, we find that firms use R&D and advertising to differentiate themselves from competitors and increase their profitability. Our results are consistent with Sutton’s (1991) theory of endogenous product differentiation. Our starting point to form new industries is to gather product descriptions from 50,673 firm annual 10-Ks filed with the Securities and Exchange Commission using web crawling algorithms. We then process the text in these product descriptions to calculate new industry classifications based on how firms are related to each other. A key advantage of this framework is that firms must file a 10-K in each year, allowing us to build classifications that change over time. The framework also provides a continuous measures of product similarity between firms both within and across industries. These tools enable us to examine how industry structure changes over time, how firms react to dynamic changes within and around their product markets, and how firms take actions to create product differentiation and barriers to entry. We measure the similarity of products offered by every unique pair of publicly traded firms every year over a comprehensive ten year sample. The vector representations of the text in each firm’s product description generate a Hotelling-like product location space for U.S. firms. The intuition behind our paper is that a firm and its rivals have locations that are very near, as their product descriptions use a common vocabulary. Analogously, firms with unrelated products are far from each other in this space. We apply clustering analysis over these locations to generate industry classifications. We compare our new industry classifications to existing classifications, and find that our industries generate superior R in explaining the cross section of firm charChamberlin (1933) and Hotelling (1929) famously show that product differentiation is fundamental to profitability and theories of industrial organization, and also that product markets can be viewed as having a spatial representation that accounts for product differentiation.