FSprof: An In-Kernel File System Operations Profiler

Developing efficient file systems is difficult. Often, profiling tools are useful for analyzing system bottlenecks and correcting them. Whereas there are several techniques to profile system call activity or disk-block activity, there are no good tools to profile file systems—which logically reside below system calls and above disk drivers. We developed a tool called FSprof that instruments existing file systems’ source code to profile their activity. This instrumentation incurs negligible runtime overhead. For file systems that do not have source code available, we also developed a thin file-system wrapper. When a profiled file system runs, it records operation frequencies and precise latencies and sorts them into configurable exponential buckets. We wrote additional tools to help verify, analyze, and display the profiling data. We ran FSprof on several popular Linux file systems: Ext2, Ext3, Reiserfs, and a stackable (layered) file system called Wrapfs. Our analysis revealed interesting discoveries about file systems and benchmarks. We analyzed bi-modal and even tri-modal distributions we found in certain operation latencies, which result from complex interactions between file system caches and disks. We illustrate how simple file system designs can lead to serious lock contention and slow down the entire operating system. We show how seemingly similar file system benchmarks can unexpectedly behave rather differently. We also observed that a tiny percentage of certain calls can have a disproportionate overall effect. FSprof is the first tool specifically designed for analyzing file system behavior, using high precision and a fine level of detail. FSprof helps developers collect and organize information, then diagnose and optimize file system performance.