Dynamic Text-Based Industry Classifications and Endogenous Product Differentiation

We study how firms differ from their competitors using new dynamic measures of product differentiation based on novel text based analysis of 50,673 product descriptions from firm 10-K statements filed yearly with the Securities Exchange Commission. This year-by-year set of firm product differentiation measures allows us to generate a set of dynamic industry classifications and new measures of industry structure and competition. Competitiveness and market structure measures based on these new classifications better correlate with firm profitability than do classifications based on SIC or NAICs. Using these new dynamic industry classifications, we examine endogenous product differentiation. We show that firms use R&D and advertising to differentiate themselves from competitors and increase their profitability. ∗University of Maryland and University of Maryland and National Bureau of Economic Research respectively. Hoberg can be reached at [email protected] and Phillips can be reached at [email protected]. We especially thank Dan Kovenock, Steve Martin, John Sutton and seminar participants at HEC, IFN (Stockholm), Insead, ISTCE (Lisbon), London Business School, Stockholm School of Economics, University of Amsterdam and University of Vienna for helpful comments. All errors are the authors alone. Copyright c ©2009 by Gerard Hoberg and Gordon Phillips. All rights reserved. Defining industry boundaries and industry competitiveness is central to the study of industrial organization. It is also central to broader disciplines in Economics and Finance, where the study of industries, or the need to control for industry, is pervasive. Our paper is based on the premise that product similarity is core to classifying industries, and that empirical work can benefit from the ability to measure industry memberships and product differentiation in every year. Using new dynamic industry classifications, we find that firms use R&D and advertising to differentiate themselves from competitors and increase their profitability. Our results are consistent with Sutton’s (1991) theory of endogenous product differentiation. Our starting point to form new industries is to gather product descriptions from 50,673 firm annual 10-Ks filed with the Securities and Exchange Commission using web crawling algorithms. We then process the text in these product descriptions to calculate new industry classifications based on how firms are related to each other. A key advantage of this framework is that firms must file a 10-K in each year, allowing us to build classifications that change over time. The framework also provides a continuous measures of product similarity between firms both within and across industries. These tools enable us to examine how industry structure changes over time, how firms react to dynamic changes within and around their product markets, and how firms take actions to create product differentiation and barriers to entry. We measure the similarity of products offered by every unique pair of publicly traded firms every year over a comprehensive ten year sample. The vector representations of the text in each firm’s product description generate a Hotelling-like product location space for U.S. firms. The intuition behind our paper is that a firm and its rivals have locations that are very near, as their product descriptions use a common vocabulary. Analogously, firms with unrelated products are far from each other in this space. We apply clustering analysis over these locations to generate industry classifications. We compare our new industry classifications to existing classifications, and find that our industries generate superior R in explaining the cross section of firm charChamberlin (1933) and Hotelling (1929) famously show that product differentiation is fundamental to profitability and theories of industrial organization, and also that product markets can be viewed as having a spatial representation that accounts for product differentiation.