On Consensus-Optimality Trade-offs in Collaborative Deep Learning
نویسندگان
چکیده
In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality . this paper, we build on recent algorithmic progresses in deep learning to explore various consensus-optimality trade-offs over fixed communication topology. First, propose the incremental -based stochastic gradient descent (i-CDSGD) algorithm, which involves multiple steps (where each agent communicates information with its neighbors) within SGD iteration. Second, generalized (g-CDSGD) algorithm that enables us navigate full spectrum complete (all agree) disagreement (each converges individual model parameters). We analytically establish convergence of proposed algorithms for strongly convex nonconvex objective functions; also analyze momentum variants case. support our via numerical experiments, demonstrate significant improvements existing methods collaborative learning.
منابع مشابه
Resolution and Relevance Trade-offs in Deep Learning
Juyong Song,1, 2, 3 Matteo Marsili,3, ∗ and Junghyo Jo1, 2, 4, † Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk 37673, Korea Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Korea The Abdus Salam International Centre for Theoretical Physics, Strada Costiera 11, 34014 Trieste, Italy School of Computational Sciences, Korea Institute for ...
متن کاملPartial Interval Set Cover - Trade-Offs between Scalability and Optimality
Given an interval I = {1, 2, ..., n} of points, a collection I of subintervals of I and a fraction 0 ≤ r ≤ 1, we consider the following variation of partial set cover. We wish to find an optimal subset of I covering at least an r-fraction of I. While this problem is easily solved exactly in quadratic time using classical methods, we focus on developing scalable algorithms which return near-opti...
متن کاملComputational Trade-offs in Statistical Learning
Computational Trade-offs in Statistical Learning by Alekh Agarwal Doctor of Philosophy in Computer Science and the Designated Emphasis
متن کاملTrade-offs in Explanatory Model Learning
In many practical applications, accuracy of a prediction is as important as understandability of the process that leads to it. Explanatory learning emerges as an important capability of systems designed for close interaction with human users. Many generic white-box predictive model types are readily available and potentially appropriate for the task (decision trees, association rules, sub-spaci...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Frontiers in artificial intelligence
سال: 2021
ISSN: ['2624-8212']
DOI: https://doi.org/10.3389/frai.2021.573731