Transfer operators and Hankel transforms between relative trace formulas, II: Rankin–Selberg theory

The Langlands functoriality conjecture, as reformulated in the “beyond endoscopy” program, predicts comparisons between (stable) trace formulas of different groups G1,G2 for every morphism G1L→LG2 their L-groups. This conjecture can be seen a special case more general which replaces reductive by spherical varieties and formula its generalization, relative formula. goal this article precursor [11] is to demonstrate, example, existence “transfer operators” formulas, generalize scalar transfer factors endoscopy. These operators have all properties that one could expect from comparison: matching, fundamental lemma Hecke algebra, (relative) characters. Most importantly, quite surprisingly, they appear abelian nature (at least, low-rank examples considered paper), even though encompass relations non-abelian harmonic analysis. Thus, are amenable application Poisson summation order perform global comparison. Moreover, we show these transforms some structure — presently escapes our understanding entirety deformations well-understood when spaces under consideration replaced “asymptotic cones”. In second paper use Rankin–Selberg theory prove local behind Rudnick's 1990 thesis (comparing stable SL2 with Kuznetsov formula) Venkatesh's 2002 (providing proof functorial tori GL2). As it turns out, latter not completely disjoint endoscopic fact, “factors” through transfer. We also study functional equation symmetric-square L-function GL2, governed an explicit “Hankel operator” at level formula, nature. A similar standard was previously developed (in language) Jacquet.