Remote Debugging with Controllable Overhead

Remote devices, a category that includes smart-phones and consumer-level wireless routers as well as servers in a datacenter and consumer PCs, are becoming an increasingly relevant target for debugging. In particular, with operating systems like Mac OS X, Linux, and Windows Mobile being used in the most recent generation of smart-phones, the OS platform for smart-phones has become essentially the same as that for the workstations used to develop their software. Also, consumer wireless routers are increasingly using Linux as the basis of their software stack. This suggests that it should be possible to create a debugging model for errors that appear in deployment. However, two obstacles make debugging these devices interactively a challenge. First, for consumer devices the user interface itself frequently lacks even the most rudimentary debugging support, such as a shell prompt. As an aggregate, servers in a datacenter have remote management infrastructure, but this is only designed for high-level status monitoring. Second, even if the user interface includes such support, the user lacks the expertise to use a debugger effectively. As a result of these two factors alone, an interactive debugger is ineffective except in a development environment. Some form of asynchronous debugging — that is, inserting debugging code and logging its results — is necessary. In this thesis proposal, we confront the problems with asynchronous debugging in these contexts. Specifically, we address the following issues: completeness, efficiency, and versatility. Completeness means having access to as much information as possible about the execution of the program being debugged. We solve the completeness problem by modifying the GNU compiler to produce instrumented versions of programs, exposing full information about every control-flow and data-flow event. Efficiency means doing this with predictable impact on runtime performance and code size — a particular concern on devices with constrained CPU, network, and storage resources. We solve the efficiency problem by guaranteeing constant overhead and only keeping instrumented versions of functions that are currently being debugged on the device. Versatility means allowing the developer of the software to control what is being debugged. We solve the versatility problem by providing a central server or distributed server infrastructure that records the debugging requirements for the code and interacts with devices in the field, providing them with necessary code and handling the logs they produce. We present our existing work on efficient instrumentation, and then go on to propose a system that addresses the above issues in a coherent, extensiblemanner, making it possible to debug even widely-distributed devices with ease. The solutions we design will be applicable not only to simple networked devices like cellular phones and wireless routers, but to any computer systems whose users are incapable of debugging the software on their own. This work was partially made possible thanks to a Computer Systems Research NSF award (CNS-0509230) and a NSF CAREER award in the Next Generation Software program (EIA-0133589).