Direct Integral Decompositions and Multiplicities for Induced Representations of Nilpotent Lie Groups

Let A" be a Lie subgroup of the connected, simply connected nilpotent Lie group G, and let f, g be the corresponding Lie algebras. Suppose that a is an irreducible unitary representation of K. We give an explicit direct integral decomposition of IndA^c;o into irreducibles. The description uses the Kirillov orbit picture, which gives a bijection between GA and the coadjoint orbits in g* (and similarly for K A, f*). Let P: f * -» g* be the canonical projection, let ff„cf* be the orbit corresponding to a, and, for ir e G A, let 0„ c g* be the corresponding orbit. The main result of the paper says essentially that ,(i>) = 0 and inequalities piiv)> 0 (where p¡ are polynomials over R), or if it is a Boolean combination of such sets and their complements. If 5 is semialgebraic it has a stratification 3a, a partition S = Sx U • ■ • u Sm im < oo) such that (i) Each S¡ is a connected, embedded submanifold in V (manifold topology = relative topology). Received by the editors November 21, 1986. 1980 Mathematics Subject Classification (1985 Revision). Primary 22E27. ©1987 American Mathematical Society 0002-9947/87 $1.00 + $.25 per page 549 License or copyright restrictions may apply to redistribution; see http://www.ams.org/journal-terms-of-use 550 L. CORWIN. F. P. GREENLEAF AND G. GRELAUD (ii) For all x e V, there exists a neighborhood NXQ V such that Nx n S, is connected (or empty), for / = 1,2,..., m. (iii)S, n Sj~* 0 =>SiQ Sf. (iv) Each S, is semialgebraic. The main properties we need are: (v) If /: V -* V is a polynomial map (or is rational, nonsingular on S) then fiS) is semialgebraic. This is one of the main results of Tarski-Seidenberg, see [14]. (vi) If í?x, 332 are stratifications of S, there is a stratification 37 that is a refinement of both. Let k = kiS) = max{dimS,: 1 < i < m) for a stratification of S. This is independent of the stratification, by (vi). For any 3e we can define a measure v9 on 5 by taking a nonvanishing ^-dimensional volume on each ^-dimensional piece of S. Various stratifications of S give the same measure class [v]. If A e S is semialgebraic and dense in S, then dim(S\,4) < dim 5, and i>iS\A) = 0. A set is algebraic if it is the difference of Zariski-open sets, i.e. the intersection of a Zariski-closed set with a Zariski-open set. The spectrum of the induced representation may lie anywhere in G A, and does not necessarily consist of representations in general position. Thus we need a cross-section for all Ad*(G)-orbits in g*. Following Pukanszky [10], one can always partition g* into Ad*(G)-invariant layers Uew U • ■ • U Ue is then a semialgebraic set cross-sectioning all orbits in g*. We will show that the layers may be chosen so that Wi = (union of the first / layers) is a Zariski-open set in g* for 1 < / < r; in particular, the set Uen> is Zariski-open, and constitutes the generic orbits. Now P~l((V) is an irreducible algebraic variety, since this is true of <7>x c f *. Therefore, if Ue = Ueui is the first layer that meets P'l(Ox), the intersection Ue n P-l(Ox) = W, fî P~l(Ox) is Zariski-open in this variety. Let 2* = 2e n Ad*(G)F~1(Cx) be the orbit representatives for this intersection. It is not hard to see that 2X is a semialgebraic set, and so determines a unique measure class (2X, v). This class is the base space for the direct integral decomposition. In terms of Zariski-open sets in the variety P~l(@x), we can define the following parameters without reference to orbit cross-sections. Noting that F_1(6?x) is Ad*(/0-invariant, because G>x is a .rv-orbit in f *, we let r = generic (maximal) dimension of an Ad*(FJ)-orbit in P~l(& ), s = generic value of dimC^ for Ad*(G)-orbits 0,= Ad*(G)/ in g* that meet P'\0X). Clearly, 5 = dim Ad*(G)/ for / e Ue. With this definition in mind, we define the "defect" parameter