Iterative builder pattern - unpacking operators

[gabmis]

Hi @rsokl, good luck with the moving and the upcoming addition to your family.

Whenever you have the time would love to hear your thoughts on this.

In the context of the iterative builder pattern If you don't use unpacking operators in your signatures everything is dandy (almost, cf previous discussion haha). In their presence, we're wondering if things could be improved.

In the following example I have an object, a DataPipeline which accepts an unlimited number of unnamed callables. This is the design we've implemented in our codebase. The order parameter is just here to indicate the order in which the elements should be chained. We're often building our training pipelines as extensions of our testing pipeline, e.g. by adding augmentations. This is the use case described below with placeholder strings instead of real callables.

We're wondering if there's a way to make things easier by reducing the amount of extra input to the builds function needs. I'm expecting the suggestions below might be against hydra-zen's principles somehow but can't pinpoint why.

import hydra_zen
from typing import List, Callable


class DataPipeline:
    def __init__(self, order: List[str], **callables: Callable):
        pass
    

test_pipe_conf = hydra_zen.builds(DataPipeline, order=["load", "prepare"], load="<load_image>", prepare="<prepare>")

# this works but I have to grab a hold of the initial target again
train_pipe_conf = hydra_zen.builds(DataPipeline, order=["load", "prepare", "augment"], augment="<augment>", builds_bases=(test_pipe_conf,))

# suggestion 1: this would be nice, but doesn't work and implies  
train_pipe_conf = hydra_zen.builds(test_pipe_conf, order=["load", "prepare", "augment"], augment="<augment>", builds_bases=(test_pipe_conf,))

# suggestion 2: this would be even nicer
train_pipe_conf = hydra_zen.builds(test_pipe_conf, order=["load", "prepare", "augment"], augment="<augment>")

Thanks again for your work,

Gabriel

[rsokl]

Hi Gabriel,

Thanks for the write-up and for the well-wishes!

I don't think I will be adding additional iterative-config supports to builds itself. builds is already quite overloaded in terms of its features; the more I add, the harder the function becomes to reason about, test, and debug.

Because you are making such heavy use of this iterative-builds pattern, I would recommend creating your own abstraction over builds. Something like:

from hydra_zen import get_target
from hydra_zen.typing import Builds, SupportedPrimitive
from typing import TypeVar, Any

T = TypeVar("T")

def iter_builds(
    cfg: type[Builds[T]], *a: SupportedPrimitive, **k: SupportedPrimitive
) -> type[Builds[T]]:

    return hydra_zen.builds(get_target(cfg), *a, **k, builds_bases=(x,))  # type: ignore

Here, if you'd like, you could modify the logic here to forego inheritance and instead inspect cfg for things like zen-meta fields, and add logic for passing those fields through. If you hit issues with this it will be easier for me to provide specific guidance with this than trying to implement this in builds itself.

Ultimately, I think this will be the shortest path to you getting the sort of ergonomics that best serve you and your team. E.g., this works now:

iter_builds(
    test_pipe_conf, order=["load", "prepare", "augment"], augment="<augment2>"
) 

[gabmis]

Hi Ryan,
Sounds like the right way to go. We're most likely going to implement something like what you suggested. Thank you very much and sorry for the delay! I'll close this now :)