Object Classification using RGB-D data for Vision Aids Apples and Oranges

With the increased availability of cheap and reliable depth sensors, imaging systems can now use depth information to better detect and locate objects in a scene. New augmented reality (AR) systems, such as the Microsoft HoloLens, are now mounting depth sensors on their glasses to improve functionality. The motivation behind our project is to perform object recognition using RGB-D (color and depth) data for a vision aid being developed at Stanford. This vision aid consists of a pair of AR goggles with an Asus Xtion depth sensor mounted on top of it. We would like to use the RGB-D data from this sensor to train a classifier that can recognize household items from a database. Information on the classified object can then be relayed to a user through the vision aid. This system has the potential to help the visually impaired navigate and perform everyday tasks more efficiently. Our training data consists of an RGB-D data set from a team at the University of Washington. We used two different 3D descriptors to extract feature vectors that describe each frame of RGB-D data. The input features to our algorithm are these descriptor vectors. We then use SVM to train a classifier that can output the predicted class of new RGB-D images. Using test data we collected with our own experimental setup, we evaluated the effectiveness of our classifier. The best cross validation results gave 89% accuracy while using the SHOTCOLOR descriptors and a subset of tested items. Experimental results from this model had a prediction accuracy of 83% when presented with testing data obtained from our experimental setup.