Learning to Remember Rare Events

نویسندگان

  • Lukasz Kaiser
  • Ofir Nachum
  • Aurko Roy
  • Samy Bengio
چکیده

Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Different Learning Levels in Multiple-choice and Essay Tests: Immediate and Delayed Retention

    This study investigated the effects of different learning levels, including Remember an Instance (RI), Remember a Generality (RG), and Use a Generality (UG) in multiple-choice and essay tests on immediate and delayed retention. Three-hundred pre-intermediate students participated in the study. Reading passages with multiple-choice and essay questions in different levels of learning were giv...

متن کامل

Carcinoma of the Breast Presenting As Deep Vein Thrombosis of the Axillary Vein

Background: Breast cancer is the most common type of malignancy in the female population, rarely is reported by thromboembolic events during the course of the disease. In this case, we present a rare case of breast cancer presenting as deep vein thrombosis of the axillary vein.Case Presentation: A 40-year- old female was admitted due to pain and swelling of her right arm and forearm of two mont...

متن کامل

Statistical Monitoring of Rare Events in Healthcare

  There are miscellaneous quality characteristics in healthcare which are interested to be monitored. However, monitoring each characteristic needs a special statistical method. There are some characteristics with very small incidence rates that it’s usually considered not to be necessary to monitoring them, since their incidence rates are so small that p and np charts are not able to monitor t...

متن کامل

Learning to Predict Rare Events in Event Sequences

Learning to predict rare events from sequences of events with categorical features is an important, real-world, problem that existing statistical and machine learning methods are not well suited to solve. This paper describes timeweaver, a genetic algorithm based machine learning system that predicts rare events by identifying predictive temporal and sequential patterns. Timeweaver is applied t...

متن کامل

Encoding of episodic information through fast task-irrelevant perceptual learning

The mechanisms guiding our learning and memory processes are of key interest to human cognition. While much research shows that attention and reinforcement processes help guide the encoding process, there is still much to know regarding how our brains choose what to remember. Recent research of task-irrelevant perceptual learning (TIPL) has found that information presented coincident with impor...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

عنوان ژورنال:
  • CoRR

دوره abs/1703.03129  شماره 

صفحات  -

تاریخ انتشار 2017