Propeller: A Scalable Metadata Organization for A Versatile Searchable File System

The exponentially increasing amount of data in file systems has made it increasingly important for file systems to provide fast file-search services. The quality of the file-search services is significantly affected by the file-index overhead, the file-search responsiveness and the accuracy of search results. Unfortunately, the existing file-search solutions either are so poorly scalable that their performance degrades unacceptably when the systems scale up, or incur so much crawling delays that they produce acceptably inaccurate results. We believe that the time is ripe for the re-designing of a searchable file system capable of accurate and scalable system-level file search. The main challenge facing the design and implementation of such a searchable file system is how to update file indices in a real-time and scalable way to obtain accurate file-search results. Thus we propose a lightweight and scalable metadata organization, Propeller, for the envisioned searchable file system. Propeller partitions the namespace according to file-access patterns, which exposes massive parallelism for the emerging manycore architecture to support future searchable file systems. The extensive evaluation results of our Propeller prototype show that it achieves significantly better file-indexing and file-search performance (up to 250×) than a centralized solution (MySQL) and only incurs negligible overhead (< 16%) to the normal file I/O operations and faster direct access performance (16.6×) on a state-ofthe-art file system (Ext4). Furthermore, the 100% recall accuracy ensures Propeller offering a feasible metadata scheme for the system-level file-search services.