python - Hydra instantiation of tuple resolved from OmegaConf custom resolver fails for nested config files

Question

OmegaConf allows you to register a custom resolver. Here is an example of resolving a tuple.

def resolve_tuple(*args):
    return tuple(args)

OmegaConf.register_new_resolver("tuple", resolve_tuple)

This can be used to resolve a value in a config file with a structure like ${tuple:1,2} to a tuple (1, 2). Along with hydra.utils.instantiate this can be used to create objects that contain or utilize tuples. For example:

config.yaml

obj:
  tuple: ${tuple:1,2}

test.py

import hydra
import hydra.utils as hu

from omegaconf import OmegaConf


def resolve_tuple(*args):
    return tuple(args)


OmegaConf.register_new_resolver('tuple', resolve_tuple)


@hydra.main(config_path='conf', config_name='config_test')
def main(cfg):
    obj = hu.instantiate(cfg.obj, _convert_='partial')
    print(obj)


if __name__ == '__main__':
    main()

Running this example returns:

$ python test.py
{'tuple': (1, 2)}

However, imagine you had a much more complex config structure. You may want to use interpolation to bring in configs from other files like so.

tuple/base.yaml

tuple: ${tuple:1,2}

config.yaml

defaults:
  - tuple: base
  - _self_

obj:
  tuple: ${tuple}

Running this example you get an error:

$ python test.py
Error executing job with overrides: []
Traceback (most recent call last):
  File "test.py", line 16, in main
    obj = hu.instantiate(cfg.obj, _convert_='partial')
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 175, in instantiate
    OmegaConf.resolve(config)
omegaconf.errors.UnsupportedValueType: Value 'tuple' is not a supported primitive type

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

The full traceback from hydra is:

Error executing job with overrides: []
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    main()
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/main.py", line 52, in decorated_main
    config_name=config_name,
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 378, in _run_hydra
    lambda: hydra.run(
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 211, in run_and_report

    return func()
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/utils.py", line 381, in <lambda>
    overrides=args.overrides,
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/hydra.py", line 111, in run
    _ = ret.return_value
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "test.py", line 17, in main
    model = hu.instantiate(cfg.obj, _convert_='partial')
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 175, in instantiate
    OmegaConf.resolve(config)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 792, in resolve
    omegaconf._impl._resolve(cfg)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 19, in _resolve_container_value
    _resolve(resolved)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 40, in _resolve
    _resolve_container_value(cfg, k)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/_impl.py", line 23, in _resolve_container_value
    node._set_value(resolved._value())
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/nodes.py", line 44, in _set_value
    self._val = self.validate_and_convert(value)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/nodes.py", line 57, in validate_and_convert
    return self._validate_and_convert_impl(value)
  File "/Users/me/anaconda3/envs/my_env/lib/python3.7/site-packages/omegaconf/nodes.py", line 134, in _validate_and_convert_impl
    f"Value '{t.__name__}' is not a supported primitive type"
omegaconf.errors.UnsupportedValueType: Value 'tuple' is not a supported primitive type

If you really dig around in the omegaconf code in the trace you will find that there is a flag for the config object allow_objects that is True in the example that passes and None in the example that does not. What is interesting is that in the _instantaite2.py file just before calling Omegaconf.resolve(config) several flags are set, one being allow_objects as True.

Is the intended behavior for these interpolated/resolved values populated from separate files to override this flag? If so, is there some way to ensure that the allow_objects flag is (or remains) true for all resolved and interpolated values?

标签： pythonfb-hydraomegaconf

Answer 1

我认为有些混乱，因为您将这个词tuple用于多种不同的目的:)

这是一个对我有用的例子：

# my_app.py
import hydra
import hydra.utils as hu

from omegaconf import OmegaConf

def resolve_tuple(*args):
    return tuple(args)

OmegaConf.register_new_resolver('as_tuple', resolve_tuple)

@hydra.main(config_path='conf', config_name='config')
def main(cfg):
    obj = hu.instantiate(cfg.obj, _convert_='partial')
    print(obj)

if __name__ == '__main__':
    main()

# conf/config.yaml
defaults:
  - subdir: base
  - _self_

obj:
  a_tuple: ${subdir.my_tuple}

# conf/subdir/base.yaml
my_tuple: ${as_tuple:1,2}

$ python my_app.py  # at the command line:
{'a_tuple': (1, 2)}

这里的主要区别是我们有a_tuple: ${subdir.my_tuple}而不是a_tuple: ${my_tuple}.

笔记：

• 元组可能会在未来某个时候被 OmegaConf 支持为一等类型。这是相关问题：https ://github.com/omry/omegaconf/issues/392
• 您提到的allow_objects标志是无证的，它的行为可能会发生变化。