Benign interpolation of noise in deep learning

Authors

DOI:

https://doi.org/10.18489/sacj.v32i2.833

Abstract

The understanding of generalisation in machine learning is in a state of flux, in part due to the ability of deep learning models to interpolate noisy training data and still perform appropriately on out-of-sample data, thereby contradicting long-held intuitions about the bias-variance tradeoff in learning. We expand upon relevant existing work by discussing local attributes of neural network training within the context of a relatively simple framework. We describe how various types of noise can be compensated for within the proposed framework in order to allow the deep learning model to generalise in spite of interpolating spurious function descriptors. Empirically, we support our postulates with experiments involving overparameterised multilayer perceptrons and controlled training data noise. The main insights are that deep learning models are optimised for training data modularly, with different regions in the function space dedicated to fitting distinct types of sample information. Additionally, we show that models tend to fit uncorrupted samples first. Based on this finding, we propose a conjecture to explain an observed instance of the epoch-wise double-descent phenomenon. Our findings suggest that the notion of model capacity needs to be modified to consider the distributed way training data is fitted across sub-units.

Author Biographies

Marthinus Wilhelmus Theunissen, Multilingual Speech Technologies, North-West University, South Africa

North-West University, South Africa

Electric, Electronic & Computer Engineering department

PhD student

Researcher

Marelie H. Davel, Multilingual Speech Technologies, North-West University, South Africa

  North-West University, South Africa Electric, Electronic & Computer Engineering department Professor   Multilingual Speech Technologies (MuST) Director

Etienne Barnard, Multilingual Speech Technologies, North-West University, South Africa

North-West University, South Africa Electric, Electronic & Computer Engineering department Professor Researcher

Downloads

Published

2020-12-08

Issue

Section

Research Papers (general)