Deep Learning Tuning Playbook

This is not an officially supported Google product.

Varun Godbole^†, George E. Dahl^†, Justin Gilmer^†, Christopher J. Shallue^‡, Zachary Nado^†

† Google Research, Brain Team

‡ Harvard University

Table of Contents

• Who is this document for?
• Why a tuning playbook?
• Guide for starting a new project
  • Choosing the model architecture
  • Choosing the optimizer
  • Choosing the batch size
  • Choosing the initial configuration
• A scientific approach to improving model performance
  • The incremental tuning strategy
  • Exploration vs exploitation
  • Choosing the goal for the next round of experiments
  • Designing the next round of experiments
  • Determining whether to adopt a training pipeline change or hyperparameter configuration
  • After exploration concludes
• Determining the number of steps for each training run
  • Deciding how long to train when training is not compute-bound
  • Deciding how long to train when training is compute-bound
• Additional guidance for the training pipeline
  • Optimizing the input pipeline
  • Evaluating model performance
  • Saving checkpoints and retrospectively selecting the best checkpoint
  • Setting up experiment tracking
  • Batch normalization implementation details
  • Considerations for multi-host pipelines
• FAQs
• Acknowledgments
• Citing
• Contributing

Who is this document for?

This document is for engineers and researchers (both individuals and teams) interested in maximizing the performance of deep learning models. We assume basic knowledge of machine learning and deep learning concepts.

Our emphasis is on the process of hyperparameter tuning. We touch on other aspects of deep learning training, such as pipeline implementation and optimization, but our treatment of those aspects is not intended to be complete.

We assume the machine learning problem is a supervised learning problem or something that looks a lot like one (e.g. self-supervised). That said, some of the prescriptions in this document may also apply to other types of problems.

Why a tuning playbook?

Currently, there is an astonishing amount of toil and guesswork involved in actually getting deep neural networks to work well in practice. Even worse, the actual recipes people use to get good results with deep learning are rarely documented. Papers gloss over the process that led to their final results in order to present a cleaner story, and machine learning engineers working on commercial problems rarely have time to take a step back and generalize their process. Textbooks tend to eschew practical guidance and prioritize fundamental principles, even if their authors have the necessary experience in applied work to provide useful advice. When preparing to create this document, we couldn't find any comprehensive attempt to actually explain how to get good results with deep learning. Instead, we found snippets of advice in blog posts and on social media, tricks peeking out of the appendix of research papers, occasional case studies about one particular project or pipeline, and a lot of confusion. There is a vast gulf between the results achieved by deep learning experts and less skilled practitioners using superficially similar methods. At the same time, these very experts readily admit some of what they do might not be well-justified. As deep learning matures and has a larger impact on the world, the community needs more resources covering useful recipes, including all the practical details that can be so critical for obtaining good results.

We are a team of five researchers and engineers who have worked in deep learning for many years, some of us since as early as 2006. We have applied deep learning to problems in everything from speech recognition to astronomy, and learned a lot along the way. This document grew out of our own experience training neural networks, teaching new machine learning engineers, and advising our colleagues on the practice of deep learning. Although it has been gratifying to see deep learning go from a machine learning approach practiced by a handful of academic labs to a technology powering products used by billions of people, deep learning is still in its infancy as an engineering discipline and we hope this document encourages others to help systematize the field's experimental protocols.

This document came about as we tried to crystalize our own approach to deep learning and thus it represents the opinions of the authors at the time of writing, not any sort of objective truth. Our own struggles with hyperparameter tuning made it a particular focus of our guidance, but we also cover other important issues we have encountered in our work (or seen go wrong). Our intention is for this work to be a living document that grows and evolves as our beliefs change. For example, the material on debugging and mitigating training failures would not have been possible for us to write two years ago since it is based on recent results and ongoing investigations. Inevitably, some of our advice will need to be updated to account for new results and improved workflows. We do not know the optimal deep learning recipe, but until the community starts writing down and debating different procedures, we cannot hope to find it. To that end, we would encourage readers who find issues with our advice to produce alternative recommendations, along with convincing evidence, so we can update the playbook. We would also love to see alternative guides and playbooks that might have different recommendations so we can work towards best practices as a community. Finally, any sections marked with a 🤖 emoji are places we would like to do more research. Only after trying to write this playbook did it become completely clear how many interesting and neglected research questions can be found in the deep learning practitioner's workflow.

Guide for starting a new project

Many of the decisions we make over the course of tuning can be made once at the beginning of a project and only occasionally revisited when circumstances change.

Our guidance below makes the following assumptions:

• Enough of the essential work of problem formulation, data cleaning, etc. has already been done that spending time on the model architecture and training configuration makes sense.
• There is already a pipeline set up that does training and evaluation, and it is easy to execute training and prediction jobs for various models of interest.
• The appropriate metrics have been selected and implemented. These should be as representative as possible of what would be measured in the deployed environment.

Choosing the model architecture

Summary: When starting a new project, try to reuse a model that already works.

• Choose a well established, commonly used model architecture to get working first. It is always possible to build a custom model later.
• Model architectures typically have various hyperparameters that determine the model's size and other details (e.g. number of layers, layer width, type of activation function).
  • Thus, choosing the architecture really means choosing a family of different models (one for each setting of the model hyperparameters).
  • We will consider the problem of choosing the model hyperparameters in Choosing the initial configuration and A scientific approach to improving model performance.
• When possible, try to find a paper that tackles something as close as possible to the problem at hand and reproduce that model as a starting point.

Choosing the optimizer

Summary: Start with the most popular optimizer for the type of problem at hand.

• No optimizer is the "best" across all types of machine learning problems and model architectures. Even just comparing the performance of optimizers is a difficult task. 🤖
• We recommend sticking with well-established, popular optimizers, especially when starting a new project.
  • Ideally, choose the most popular optimizer used for the same type of problem.
• Be prepared to give attention to *all* hyperparameters of the chosen optimizer.
  • Optimizers with more hyperparameters may require more tuning effort to find the best configuration.
  • This is particularly relevant in the beginning stages of a project when we are trying to find the best values of various other hyperparameters (e.g. architecture hyperparameters) while treating optimizer hyperparameters as nuisance parameters.
  • It may be preferable to start with a simpler optimizer (e.g. SGD with fixed momentum or Adam with fixed $\epsilon$, $\beta_{1}$, and $\beta_{2}$) in the initial stages of the project and switch to a more general optimizer later.
• Well-established optimizers that we like include (but are not limited to):
  • SGD with momentum (we like the Nesterov variant)
  • Adam and NAdam, which are more general than SGD with momentum. Note that Adam has 4 tunable hyperparameters and they can all matter!
    • See How should Adam's hyperparameters be tuned?

Choosing the batch size

Summary: The batch size governs the training speed and shouldn't be used to directly tune the validation set performance. Often, the ideal batch size will be the largest batch size supported by the available hardware.

• The batch size is a key factor in determining the training time and computing resource consumption.
• Increasing the batch size will often reduce the training time. This can be highly beneficial because it, e.g.:
  • Allows hyperparameters to be tuned more thoroughly within a fixed time interval, potentially resulting in a better final model.
  • Reduces the latency of the development cycle, allowing new ideas to be tested more frequently.
• Increasing the batch size may either decrease, increase, or not change the resource consumption.
• The batch size should not be treated as a tunable hyperparameter for validation set performance.
  • As long as all hyperparameters are well-tuned (especially the learning rate and regularization hyperparameters) and the number of training steps is sufficient, the same final performance should be attainable using any batch size (see Shallue et al. 2018).
  • Please see Why shouldn't the batch size be tuned to directly improve validation set performance?

Determining the feasible batch sizes and estimating training throughput

Choosing the batch size to minimize training time

Choosing the batch size to minimize resource consumption