Statistical Analysis for Multisite Trials Using Instrumental Variables With Random Coefficients

Multisite trials can clarify the average impact of a new program and the heterogeneity of impacts across sites. Unfortunately, in many applications, compliance with treatment assignment is imperfect. For these applications, we propose an instrumental variable (IV) model with person-specific and site-specific random coefficients. Site-specific IV coefficients can be interpreted as site-average effects of program participation or as site-average effects of participation for the compliers. The validity of these interpretations depends on the analyst’s assumptions. Within the framework of a two-level hierarchical linear model, we propose three ways to estimate the mean and variance of these site-specific effects: (a) estimate the impact of program participation and its standard error in each site, then combine these site-specific statistics to estimate the mean and variance of the true site effects; (b) estimate the mean and variance of the effect of treatment assignment on the outcome and the mean and variance of the effect of treatment assignment on program participation; then combine these results to obtain estimates of the mean and variance of the effect of program participation; and (c) use Site by Treatment interactions as multiple instruments. If we assume the IV coefficients to be homogenous across sites, the three approaches are equivalent to variants of familiar two-stage least squares estimates with site fixed effects. Estimates based on our model are valid under a weaker assumption: that site-average levels of compliance are independent of site-average effects of program participation. To illustrate our approach, we evaluate a district-wide policy intended to increase math instructional time in math for low-achieving students. Finally, we discuss how Method (c) can be extended to incorporate multiple mediators.