Call Tree Reversal is NP-Complete

The data-flow of a numerical program is reversed in its adjoint. We discuss the combinatorial optimization problem that aims to find optimal checkpointing schemes at the level of call trees. For a given amount of persistent memory the objective is to store selected arguments and/or results of subroutine calls such that the overall computational effort (the total number of floating-point operations performed by potentially repeated forward evaluations of the program) of the data-flow reversal is minimized. CALL TREE REVERSAL is shown to be NP-complete. 1 Background We consider implementations of multi-variate vector functions F : IRn → IRm as computer programs y = F(x). The interpretation of reverse mode automatic differentiation (AD) [8] as a semantic source code transformation performed by a compiler yields an adjoint code x̄+ = F̄(x, ȳ). For given x and ȳ the vector x̄ is incremented with (F ′(x))T · ȳ where F ′(x) denotes the Jacobian matrix of F at x. Adjoint codes are of particular interest for the evaluation of large gradients as the complexity of the adjoint computation is independent of the gradient’s size. Refer to [1–4] for an impressive collection of applications where adjoint codes are instrumental to making the transition form pure numerical simulation to optimization of model parameters or even of the model itself. In this paper we propose an extension to the notion of joint call tree reversal [8] with the potential storage of the results of a subroutine call. We consider call trees as runtime representations of the interprocedural flow of control of a program. Each node in a call tree corresponds uniquely to a subroutine call.1 We assume that no checkpointing is performed at the intraprocedural level, that is, a “store-all” strategy is employed inside all subroutines. A graphical notation for call tree reversal under the said constraints is proposed in Figure 1. A given subroutine can be executed without modifications (“advance”) or in an augmented form where all values that are required for the evaluation of its adjoint are stored (taped) on appropriately typed stacks (“tape (store all)”). We refer to this memory as the tape associated with a subroutine call, not to be confused with the kind of tape as generated by AD-tools that use operator overloading such as ADOL-C [9] or variants of the differentiation-enabled NAGWare Fortran compiler [14]. The arguments of a subroutine call can be stored (“store arguments”) and restored (“restore arguments”). Results of a subroutine call can be treated similarly (“store results” and “restore results”). The adjoint propagation yields the reversed data-flow due to popping the previously pushed values from the corresponding stacks (“reverse (store all)”). Subroutines that only call other subroutines without performing any local computation are represented by “dummy calls.” For example, such wrappers can be used to visualize arbitrary checkpointing schemes for time evolutions (implemented as loops whose body 1Generalizations may introduce nodes for various parts of the program, thus yielding arbitrary checkpointing schemes. is wrapped into a subroutine). Moreover they occur in the reduction used for proving CALL TREE REVERSAL to be NP-complete. Dummy calls can be performed in any of the other seven modes. advance tape (store all) ? store arguments 6 restore arguments ? store results 6 restore results reverse (store all) dummy call Fig. 1. Calling modes for interprocedural data-flow reversal. Figure 2 illustrates the reversal in split (b), classical joint (c), and joint with result checkpointing (d) modes for the call tree in (a). The order of the calls is from left to right and depth-first.