Standing on the Shoulders of Others: Using Proxies to Opportunistically Boost Distributed Applications

Distributed applications are increasingly relying on and integrating remote resources including community data sources, services, and computational platforms. The coupling of distributed resources for data-intensive applications can expose bottlenecks across highly shared pathways, e.g. the network. In this paper, we consider applications that access remote clouds for data, computation, or services, and focus on alleviating the bottlenecks that they may encounter by using middleware deployed on a proxy network. We propose a proxy network that sits between the cloud and the end-user application offering resources to mitigate bottlenecks. In particular, we show how proxies can eliminate network bottlenecks by smart routing and perform in-network computations to boost application performance. We build on existing literature to create a unique synthesis of ideas that is the proxy network. Our results obtained through experiments on PlanetLab are promising, showing substantial improvement in latency, bandwidth, and even jitter, which can benefit a wide class of data-intensive applications. Our microbenchmarks show that routing data through select proxies can accelerate network transfer by at least 25% in almost half the cases considered. Experiments using a distributed Montage workflow mapped on a WAN showed 13% end-to-end performance improvement when data was routed through a proxy network. In addition, proxies also reduced the montage output data delivery time to users on mobile devices by 65-80% through compression and data transformation. ⋆ The authors would like to acknowledge grant NSF/IIS-0916425 that supported this research 1 This research made use of Montage, funded by the National Aeronautics and Space Administration’s Earth Science Technology Office, Computation Technologies Project, under Cooperative Agreement Number NCC5-626 between NASA and the California Institute of Technology. Montage is maintained by the NASA/IPAC Infrared Science Archive.