Neural network immediately overfitting
Neural network immediately overfitting I have a FFNN with 2 hidden layers for a regression task that overfits almost immediately (epoch 2-5, depending on # hidden units). (ReLU, Adam, MSE, same # hidden units per layer, tf.keras) 32 neurons: 128 neurons: I will be tuning the number of hidden units, but to limit the search space I would like to know what the upper and lower bounds should be. Afaik it is better to have a too large network and try to regularize via L2-reg or dropout than to lower the network's capacity -- because a larger network will have more local minima, but the actual loss value will be better. Is there any point in trying to regularize (via e.g. dropout) a network that overfits from the get-go? If so I suppose I could increase both bounds. If not I would lower them. model = Sequential() model.add(Dense(n_neurons, 'relu')) model.add(Dense(n_neurons, 'relu')) model.add(Dense(1, 'linear')) model.compile('adam', 'mse') ...