We face two major problems while tuning hyperparameters:
Memory constraint : Sometimes we have to deal with hundreds of gigabytes of data. We cannot store such a huge amount of data in RAM. While training a neural network, we send data in batches. One of the possible solutions is larger memory, which is not feasible always.
Time/computation constraint : Let’s say our data fits into memory, but we are training a deep neural network (DNN) or there is a huge search space for hyperparameter optimization. This can consume a great amount of time.
