Debugging hygienic macros

Over the past two decades, Scheme macros have evolved into a powerful API for the compiler front-end. Like Lisp macros, their predecessors, Scheme macros expand source programs into a small core language; unlike Lisp systems, Scheme macro expanders preserve lexical scoping, and advanced Scheme macro systems handle other important properties such as source location. Using such macros, Scheme programmers now routinely develop the ultimate abstraction: embedded domain-specific programming languages. Unfortunately, a typical Scheme programming environment provides little support for macro development. This lack makes it difficult for programmers to debug their macros and for novices to study the behavior of macros. In response, we have developed a stepping debugger specialized to the concerns of macro expansion. This debugger presents the macro expansion process as a linear rewriting sequence of annotated terms; it graphically illustrates the binding structure of the program as expansion reveals it; and it adapts to the programmer’s level of abstraction, hiding details of syntactic forms that the programmer considers built-in. 1. The Power of Macros Modern programming languages support a variety of abstraction mechanisms: higher-order functions or delegates, class systems, expressive type systems, module systems, and many more. With these, programmers can develop code for reuse; establish single points of control for a piece of functionality; decouple distinct components and work on them separately. As Paul Hudak [21] has argued, however, “the ultimate abstraction is a . . . language.” The ideal programming language should therefore allow programmers to develop, encapsulate, and reuse entire sub-languages. The Lisp and Scheme family of languages empower programmers to do just that. Through macros, they offer the programmer the ability to define new forms of expressions and definitions with custom behavior. These syntactic abstractions may introduce new binding positions, perform analyses on their subterms, and change the context of their subexpressions. In some systems, macros can share information, perform sophisticated computation, and detect and report errors, all at compile time. As some Scheme implementers have put it, macros have become a true API to the front-end of the compiler. Macro definitions are a uniform mechanism for extending the expression and definition forms of the language. Programmers can mix extensions freely using Scheme’s scoping mechanisms, such as modules and local bindings, rather than needing preprocessors, specialized compilers, or program generators. If attached to an expressive language [10] macros suffice to implement many general-purpose abstraction mechanisms as 1 This paper revises and extends a conference publication [5]. This work was partially supported by several NSF grants. Preprint submitted to Science of Computer Programming 10 June 2009 libraries. For example, programmers have used macros to extend Scheme with constructs for pattern matching [37], relations in the spirit of Prolog [9,18,29], extensible looping constructs [8,28], class systems [1,26] and component systems [6,15,35], among others. In addition, programmers have also used macros to handle metaprogramming tasks traditionally implemented outside the language using preprocessors or special compilers: Owens et al. [25] have added a parser generator library to Scheme; Sarkar et al. [27] have created an infrastructure for expressing nano-compiler passes; and Herman and Meunier [20] have used macros to improve the static analysis of Scheme programs. As a result, implementations of Scheme such as PLT Scheme [14] have a core of a dozen or so syntactic constructs but appear to implement a language as rich in features as Common Lisp [32]. To support these increasingly ambitious applications, macro systems have had to evolve, too, because true syntactic abstractions must be indistinguishable from features directly supported by the implementation’s compiler or interpreter. Historically, macros had humble beginnings in Lisp systems, where they were compile-time functions over program fragments, usually plain S-expressions. Unfortunately, these simplistic macros do not really define abstractions. For example, because Lisp macros manipulate variables by name alone, references they introduce can be captured by bindings in the context of the macro’s use. This tangling of scopes reveals implementation details instead of encapsulating them. In response, researchers developed the notions of macro hygiene [22], referential transparency [4,7], and phase separation [13] and implemented macro systems guaranteeing those properties. Now macros manipulate program representations that carry information about lexical scope; affecting scope takes deliberate effort on the part of the macro writer. Given the power of macros and the degree to which Scheme programmers benefit from them, it is astonishing that no Scheme implementation offers solid debugging support for macros. The result is that programmers routinely ask on Scheme mailing lists about unforeseen effects and subtle errors arising from a faulty or incomplete understanding of the macro system. While experts always love to come to their aid, these questions demonstrate that programmers need tools for exploring macro expansion just as they need tools for exploring ordinary program execution. In this paper, we present the first macro stepper for Scheme. In the next section we discuss the properties of macro programming that inform the design of our debugger, and in subsequent sections we discuss the implementation of the stepper, describe its foundations with a formal model, and prove those foundations correct. 2. Design Considerations for a Macro Debugger A debugger must be tailored to its target language. Debugging a logic program is fundamentally different from debugging assembly code. Most importantly, a debugger should reflect the programmer’s mental model of the language. This model determines what information the debugger displays as well as how the programmer accesses it. It determines whether the programmer inspects variable bindings resulting from unification or memory locations and whether the programmer explores a tree-shaped logical derivation or steps through line-sized statements. Following this analysis, the design of our debugger for macros is based on three principles of Scheme macros: (i) Macros are rewriting rules. (ii) Macros respect lexical scoping. (iii) Macros define layered abstractions. This section elaborates these three principles and their influence on the design of our debugger. 2.1. Macros are rewriting rules Intuitively, macro definitions specify rewriting rules, and macro expansion is the process of applying these rules to the program until all of the macros have been eliminated and the program uses only the primitive forms of the language.