“Backpropagation was the key that unlocked the power of deep learning, and deep learning is the engine that drives artificial intelligence.”
– Geoffrey Hinton
Geoffrey Hinton, David Rumelhart, & Ronald Williams published the influential 1986 paper “Learning representations by back-propagating errors.” They detailed a novel use for the backpropagation algorithm by demonstrating its effectiveness in training neural networks. This discovery changed the world.
There have been a number of key researchers in the history of backpropagation. In a 1974 PhD thesis, Paul Werbos was the first to describe the backpropagation algorithm for training multi-layer perceptrons, which laid the groundwork for Hinton and others to build on.
Stephen Grossberg introduced the Adaptive Resonance Theory in the late 1970s and early 1980s. Adaptive Resonance Theory is focused on an unsupervised learning algorithm for neural networks. Grossberg’s work contributed to a broader understanding of neural networks and their learning capabilities.
Kunihiko Fukushima proposed the Neocognitron in 1980. The neocognitron is a hierarchical, multi-layered artificial neural network used for pattern recognition. Fukushima’s design inspired the development of convolutional neural networks, which would later employ backpropagation as a key learning mechanism.
In the early 1980s, John Hopfield introduced the Hopfield network, a type of recurrent neural network that can store patterns and memories. This work is seen as reigniting researchers’ interest in neural networks. Hopfield had an important role in the research momentum that led to the popularization of the backpropagation algorithm.
Another influential neural network researcher was Teuvo Kohonen. He developed the Self-Organizing Map in the early ’80s, which is an unsupervised learning technique for neural networks. Kohonen’s work in neural networks helped build the foundation for subsequent advances, including backpropagation.
Yann LeCun is recognized for applying the backprop algorithm to convolutional neural networks in the late 1980s. This significantly advanced the field of computer vision and is considered foundational to many modern AI applications.
Sepp Hochreiter and Jürgen Schmidhuber introduced the Long Short-Term Memory architecture in 1997. This architecture uses backpropagation-through-time to train recurrent neural networks. LSTM has had a major impact on natural language processing and time series prediction tasks.
