Hydra

Integrate W&B with Hydra to manage complex configurations for ML experiments and log hyperparameters automatically.

Hydra is an open source Python framework that simplifies the development of research and other complex applications. The key feature is the ability to dynamically create a hierarchical configuration by composition and override it through config files and the command line.

This page shows how to combine Hydra-based configuration management with W&B experiment tracking, so you can keep Hydra’s composable configs while gaining W&B’s visualization, hyperparameter optimization, and run comparison capabilities. The following sections cover tracking metrics, logging hyperparameters from Hydra configs, troubleshooting multiprocessing, and optimizing hyperparameters with W&B Sweeps.

Track metrics

To send metrics from a Hydra-configured run to W&B, use wandb.init() and wandb.Run.log() as you normally would. In the following example, wandb.entity and wandb.project are defined within a Hydra configuration file so that the same config drives both Hydra and W&B.

import wandb


@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):

    with wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project) as run:
      run.log({"loss": loss})

Track hyperparameters

Logging Hydra’s configuration to W&B lets you see every hyperparameter alongside the run’s metrics, making experiments easier to compare and reproduce.

Hydra uses omegaconf as the default way to interface with configuration dictionaries. OmegaConf’s dictionary isn’t a subclass of primitive dictionaries, so directly passing Hydra’s Config to wandb.Run.config leads to unexpected results on the dashboard. Convert omegaconf.DictConfig to the primitive dict type before passing it to wandb.Run.config.

@hydra.main(config_path="configs/", config_name="defaults")
def run_experiment(cfg):
  with wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project) as run:
    run.config = omegaconf.OmegaConf.to_container(
        cfg, resolve=True, throw_on_missing=True
    )
    run = wandb.init(entity=cfg.wandb.entity, project=cfg.wandb.project)
    run.log({"loss": loss})
    model = Model(**run.config.model.configs)

Troubleshoot multiprocessing

If your process stops responding when started, the known multiprocessing issue in distributed training might be the cause. To resolve it, change W&B’s multiprocessing protocol by either adding an extra settings parameter to wandb.init():

wandb.init(settings=wandb.Settings(start_method="thread"))

or by setting a global environment variable from your shell:

export WANDB_START_METHOD=thread

Optimize hyperparameters

W&B Sweeps is a hyperparameter search platform that provides insights and visualizations for W&B experiments with minimal code overhead. Sweeps integrates with Hydra projects without requiring code changes. You only need a configuration file that describes the parameters to sweep over.

The following sweep.yaml file is an example:

program: main.py
method: bayes
metric:
  goal: maximize
  name: test/accuracy
parameters:
  dataset:
    values: [mnist, cifar10]

command:
  - ${env}
  - python
  - ${program}
  - ${args_no_hyphens}

Invoke the sweep:

wandb sweep sweep.yaml

W&B automatically creates a sweep inside your project and returns a wandb agent command. Run that command on each machine on which you want to execute the sweep.

Pass parameters not present in Hydra defaults

Hydra supports passing extra parameters through the command line that aren’t present in the default configuration file, by using a + before the command. For example, pass an extra parameter with some value by calling:

python program.py +experiment=some_experiment

You can’t sweep over such + configurations the same way you would when configuring Hydra Experiments. To work around this, initialize the experiment parameter with a default empty file and use a W&B Sweep to override those empty configs on each call. For more information, read the W&B report Configuring W&B Projects with Hydra.