Why are structured configs required for correct usage of nested config groups

[ptonner]

Apologies b/c below is quite long winded. The basic idea and question is pretty straightforward, I'm just having trouble compressing it down much further (despite hydra_zen's exceptional ability to reduce boilerplate 😄 )

Building on the ideas outlined here, I'm trying to understand why config objects generated by builds are needed for store-ing higher-level/top-level objects.

What works

To try and make this more clear, hear is an example that I adapted from the docs, which works as expected:

# ex1.py
from hydra_zen import builds, instantiate, store, zen

def bar(a: int = 1):
    print("bar", a, type(a))
    return a+1

bar_cfg = builds(bar, populate_full_signature=True)
store(bar_cfg, group="op/bar", name="default")

@store(populate_full_signature=True, zen_partial=True, group="op", bar=bar_cfg)
def foo(bar: int):
    print("foo", bar, type(bar))
    return bar * 2

@store(name="main")
def main(op):
    print(op)
    print(op())

store.add_to_hydra_store()
zen(main).hydra_main(config_name="main", config_path=None, version_base="1.3")

This works as I expect, I can call directly

python ex1.py +op=foo
> bar 1 <class 'int'>
> functools.partial(<function foo at 0x7fc2d6c1e830>, bar=2)
> foo 2 <class 'int'>
> 4

As well as supporting nested overrides

python ex1.py +op=foo op.bar.a=100
> bar 100 <class 'int'>
> functools.partial(<function foo at 0x7fdc135d6830>, bar=101)
> foo 101 <class 'int'>
> 202

What doesn't work

# ex2.py
from hydra_zen import builds, instantiate, store, zen, make_config

@store(group="op/bar", name="default", populate_full_signature=True)
def bar(a: int = 1):
    print("bar", a, type(a))
    return a+1

@store(populate_full_signature=True, zen_partial=True, group="op")
def foo(bar: int = 1):
    print("foo", bar, type(bar))
    return bar * 2

@store(name="main")
def main(op):
    print(op)
    print(op())

store.add_to_hydra_store()
zen(main).hydra_main(config_name="main", config_path=None, version_base="1.3")

Basically the differences are:

• wrap the definition with bar directly with a store decorator
• don't provide config object when storing foo

This works fine if not overriding bar

python ex2.py +op=foo
> functools.partial(<function foo at 0x7fcc29c87d00>, bar=1)
> foo 1 <class 'int'>
> 2

But when I try to run this, there seems to be an issue if I try to override bar:

python ex2.py +op=foo +op/bar=default
> In 'op/bar/default': ValidationError raised while composing config:
> Value '{'_target_': '__main__.bar', 'a': 1}' of type 'DictConfig' could not be converted to Integer
>    full_key: op.bar
>    object_type=PartialBuilds_foo

> Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Why I'm confused

I can understand the basic issue here (I think): essentially in the second case there's no alignment between the group op/bar and the argument bar for foo. Or at least, it appears hydra_zen isn't instaniating op/bar in the same way in this case. So my question is essentially: why doesn't this work?

My expectation is that it shouldn't matter whether the store call for foo is given a builds config to work correctly, but clearly it does. I figure this is either

1. An opportunity for me to better understand why this is expected, which I would appreciate any suggestions on
2. or, functionality that could actually be supported by hydra_zen

Why bother?

In case it's useful, here's a slightly expanded example of why I'm going through all this trouble in the first place. Essentially, I want to provide a variable operation CLI, with overrides providing paths to different sub-commands (which I'm calling op).

from hydra_zen import builds, instantiate, store, zen

def bar(a: int = 1):
    print("bar", a, type(a))
    return a+1

bar_cfg = builds(bar, populate_full_signature=True)
store(bar_cfg, group="op/bar", name="default")

def buz(a: float=1.0):
    print("buz", a, type(a))
    return a / 10

buz_cfg = builds(buz, populate_full_signature=True)
store(buz_cfg, group="op/buz", name="default")

op_store = store(populate_full_signature=True, zen_partial=True, group="op") 
@op_store(bar=bar_cfg)
def foo(bar: int):
    print("foo", bar, type(bar))
    return bar * 2

@op_store(buz=buz_cfg)
def baz(buz: float):
    print("baz")
    return buz * 2

@store(name="main")
def main(op):
    print(op)
    print(op())

store.add_to_hydra_store()
zen(main).hydra_main(config_name="main", config_path=None, version_base="1.3")

Then I can mix and match calls like

python full.py +op=baz op.buz.a=1
> ...

python full.py +op=foo op.bar.a=100
> ...

To be clear these all work as expected and are fine. I just wouldn't mind avoiding the extra steps of creating a separate config object for the nested bar and buz components.

Also alternatively, maybe there's just a more clear path for supporting subcommands with hydra_zen that I'm just missing or unware of?