Frobenius Recip-rocity and Extensions of Nilpotent Lie Groups

In §1 we use COO-vector methods, essentially Frobenius reciprocity, to derive the Howe-Richardson multiplicity formula for compact nilmanifolds. In §2 we use Frobenius reciprocity to generalize and considerably simplify a reduction procedure developed by Howe for solvable groups to general extensions of nilpotent Lie groups. In §3 we give an application of the previous results to obtain a reduction formula for solvable Lie groups. Introduction. In the representation theory of rings the fact that the tensor product functor and Hom functor are adjoints is of fundamental importance and at the time very simple and elegant. When this adjointness property is translated into a statement about representations of finite groups via the group ring, the classical Frobenius reciprocity theorem emerges. From this point of view, Frobenius reciprocity, aside from being a useful tool, provides a very natural explanation for many phenomena in representation theory. When G is a Lie group and r is a discrete cocompact subgroup, a main problem in harmonic analysis is to describe the decomposition of ind~(1)-the quasiregular representation. When G is semisimple the problem is enormously difficult, and little is known in general. However, if G is nilpotent, a rather detailed description of ind¥(l) is available, thanks to the work of Moore, Howe, Richardson, and Corwin and Greenleaf [M, H-I, R, C-G]. If G is solvable Howe has developed an inductive method of describing ind~(l) [H-2]. In all of the above cases, the basic tools used are the ingredients of the now standard Mackey normal subgroup analysis, and the result obtained might be viewed as saying that a version of the classical Frobenius reciprocity theorem holds in each instance. Of course, it would be desirable to obtain these results as a consequence of a Frobenius reciprocity theorem. When studying semisimple Lie groups, the authors of [G] recognize the correct formulation of Frobenius reciprocity for Lie groups and provide a proof that depends on the rigid structural aspects of semisimple Lie groups and their representations. This structural property lets one replace an analytic problem with an algebraic one that can be dealt with. In [Po], Poulsen saw what the core of the difficulty was and using elliptic regularity arguments solved the technical analytic problem in general. From here it was a quick step in proving very general Frobenius reciprocity theorems for Lie groups, as Penney did in [P-I] and [P-2]. Received by the editors May 4, 1984 and, in revised form, February 20, 1986. 1980 Mathematics Subject Classification. Primary 22E25; Secondary 22E40. 123 ©1986 American Mathematical Society 0002-9947/86 $1.00 + $.25 per page License or copyright restrictions may apply to redistribution; see http://www.ams.org/journal-terms-of-use