I’m training some models for sequence processing deep learning. Right now I’m narrowing down the hyperparameters, and I do that by running a few dozen short training models with different, randomly chosen parameters. I just ran a batch of 20. I truncate at a validation patience of 30, so when the best run of the model is more than 30 validations ago, training stops.
Around 2200, the loss function (loss bad) was at 0.800 on model 20. All losses for the next set of rounds were above 0.800, and it takes about a minute per round. I can wait a few minutes to see the end of it. I started poking at some fanfiction.
At 2230, it hit 0.792. It has to be almost done, right?
2300, 0.745.
2330, 0.715.
At 0100 I gave up and left. It was still going.