The capacity of a deep learning neural network model controls the scope of the types of mapping functions that it is able to learn.<\/p>\n
A model with too little capacity cannot learn the training dataset meaning it will underfit, whereas a model with too much capacity may memorize the training dataset, meaning it will overfit or may get stuck or lost during the optimization process.<\/p>\n
The capacity of a neural network model is defined by configuring the number of nodes and the number of layers.<\/p>\n
In this tutorial, you will discover how to control the capacity of a neural network model and how capacity impacts what a model is capable of learning.<\/p>\n
After completing this tutorial, you will know:<\/p>\n
- \n
- Neural network model capacity is controlled both by the number of nodes and the number of layers in the model.<\/li>\n
- A model with a single hidden layer and sufficient number of nodes has the capability of learning any mapping function, but the chosen learning algorithm may or may not be able to realize this capability.<\/li>\n
- Increasing the number of layers provides a short-cut to increasing the capacity of the model with fewer resources, and modern techniques allow learning algorithms to successfully train deep models.<\/li>\n<\/ul>\n
Let\u2019s get started.<\/p>\n
![\"How](\"https:\/\/machinelearningmastery.com\/wp-content\/uploads\/2019\/02\/How-to-Control-Neural-Network-Model-Capacity-With-Nodes-and-Layers.jpg\")
<\/p>\nHow to Control Neural Network Model Capacity With Nodes and Layers
Photo by Bernard Spragg. NZ<\/a>, some rights reserved.<\/p>\n<\/div>\nTutorial Overview<\/h2>\n
This tutorial is divided into five parts; they are:<\/p>\n
- \n
- Controlling Neural Network Model Capacity<\/li>\n
- Configure Nodes and Layers in Keras<\/li>\n
- Multi-Class Classification Problem<\/li>\n
- Change Model Capacity With Nodes<\/li>\n
- Change Model Capacity With Layers<\/li>\n<\/ol>\n
Controlling Neural Network Model Capacity<\/h2>\n
The goal of a neural network is to learn how to map input examples to output examples.<\/p>\n
Neural networks learn mapping functions. The capacity of a network refers to the range or scope of the types of functions that the model can approximate.<\/p>\n
\n
Informally, a model\u2019s capacity is its ability to fit a wide variety of functions.<\/p>\n<\/blockquote>\n
\u2014 Pages 111-112, Deep Learning<\/a>, 2016.<\/p>\n
A model with less capacity may not be able to sufficiently learn the training dataset. A model with more capacity can model more different types of functions and may be able to learn a function to sufficiently map inputs to outputs in the training dataset. Whereas a model with too much capacity may memorize the training dataset and fail to generalize or get lost or stuck in the search for a suitable mapping function.<\/p>\n
Generally, we can think of model capacity as a control over whether the model is likely to underfit or overfit a training dataset.<\/p>\n
\n
We can control whether a model is more likely to overfit or underfit by altering its capacity.<\/p>\n<\/blockquote>\n
\u2014 Pages 111, Deep Learning<\/a>, 2016.<\/p>\n
The capacity of a neural network can be controlled by two aspects of the model:<\/p>\n
- \n
- Number of Nodes.<\/li>\n
- Number of Layers.<\/li>\n<\/ul>\n
A model with more nodes or more layers has a greater capacity and, in turn, is potentially capable of learning a larger set of mapping functions.<\/p>\n
\n
A model with more layers and more hidden units per layer has higher representational capacity \u2014 it is capable of representing more complicated functions.<\/p>\n<\/blockquote>\n
\u2014 Pages 428, Deep Learning<\/a>, 2016.<\/p>\n
The number of nodes in a layer is referred to as the width<\/strong>.<\/p>\n
Developing wide networks with one layer and many nodes was relatively straightforward. In theory, a network with enough nodes in the single hidden layer can learn to approximate any mapping function, although in practice, we don\u2019t know how many nodes are sufficient or how to train such a model.<\/p>\n
The number of layers in a model is referred to as its depth<\/strong>.<\/p>\n
Increasing the depth increases the capacity of the model. Training deep models, e.g. those with many hidden layers, can be computationally more efficient than training a single layer network with a vast number of nodes.<\/p>\n
\n
Modern deep learning provides a very powerful framework for supervised learning. By adding more layers and more units within a layer, a deep network can represent functions of increasing complexity.<\/p>\n<\/blockquote>\n