COMS 4771 Spring 2015 Features

In many applications, no linear classifier over the “raw” set of features will perfectly separate the data. One recourse is to find additional features that are predictive of the label. This is called feature engineering, and is often a substantial part of the job of a machine learning practitioner. In some applications, it is possible to “throw in the kitchen sink”—i.e., include all possible features that might possibly be relevant. For instance, in document classification, one can include a feature for each possible word in the vocabulary that indicates whether that word is present in the given document (or counts the number of occurrences). One can also include a feature for each possible pair of consecutive words (“bi-grams”), each possible triple of consecutive words (“tri-grams”), and so on. In general, it is common to automatically generate features based on existing features x ∈ Rd, such as quadratic interaction features x 7→ (x1x2, x1x3, . . . , x1xd, x2x3, . . . , xd−1xd) ∈ R( d 2), as well as higher-order interaction features. The main drawback of these “kitchen sink” feature expansions is that it may be computationally expensive to work explicitly in the expanded feature space. Fortunately, there are some ways around this.