On Consensus-Optimality Trade-offs in Collaborative Deep Learning

نویسندگان

چکیده

In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality . this paper, we build on recent algorithmic progresses in deep learning to explore various consensus-optimality trade-offs over fixed communication topology. First, propose the incremental -based stochastic gradient descent (i-CDSGD) algorithm, which involves multiple steps (where each agent communicates information with its neighbors) within SGD iteration. Second, generalized (g-CDSGD) algorithm that enables us navigate full spectrum complete (all agree) disagreement (each converges individual model parameters). We analytically establish convergence of proposed algorithms for strongly convex nonconvex objective functions; also analyze momentum variants case. support our via numerical experiments, demonstrate significant improvements existing methods collaborative learning.

برای دانلود باید عضویت طلایی داشته باشید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Resolution and Relevance Trade-offs in Deep Learning

Juyong Song,1, 2, 3 Matteo Marsili,3, ∗ and Junghyo Jo1, 2, 4, † Asia Pacific Center for Theoretical Physics, Pohang, Gyeongbuk 37673, Korea Department of Physics, Pohang University of Science and Technology, Pohang, Gyeongbuk 37673, Korea The Abdus Salam International Centre for Theoretical Physics, Strada Costiera 11, 34014 Trieste, Italy School of Computational Sciences, Korea Institute for ...

متن کامل

Partial Interval Set Cover - Trade-Offs between Scalability and Optimality

Given an interval I = {1, 2, ..., n} of points, a collection I of subintervals of I and a fraction 0 ≤ r ≤ 1, we consider the following variation of partial set cover. We wish to find an optimal subset of I covering at least an r-fraction of I. While this problem is easily solved exactly in quadratic time using classical methods, we focus on developing scalable algorithms which return near-opti...

متن کامل

Computational Trade-offs in Statistical Learning

Computational Trade-offs in Statistical Learning by Alekh Agarwal Doctor of Philosophy in Computer Science and the Designated Emphasis

متن کامل

Trade-offs in Explanatory Model Learning

In many practical applications, accuracy of a prediction is as important as understandability of the process that leads to it. Explanatory learning emerges as an important capability of systems designed for close interaction with human users. Many generic white-box predictive model types are readily available and potentially appropriate for the task (decision trees, association rules, sub-spaci...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: Frontiers in artificial intelligence

سال: 2021

ISSN: ['2624-8212']

DOI: https://doi.org/10.3389/frai.2021.573731