Add options to make all variables available as environment variables

[jamestenglish]

I want to run something like Helmsman which takes environment variables and substitutes them into a template. I want to be able to use the variables defined in Taskvars.yml in the environment but I don't have to have to explicitly include each one like:

tasks:
  test:
    env:
      ENV1: {{.ENV1}}
      ENV2: {{.ENV2}}
    cmds:
      - helmsman

is there a way to do something like:

tasks:
  test:
    env:
      setAllVariablesToEnvironment: true
    cmds:
      - helmsman

[andreynering]

Hi @jamestenglish.

No such feature exists yet, but I'll keep this issue as an idea that can be implemented in the future.

[andreynering]

As an alternative, you could set environment variables globally using env: on your Taskfile.yml.

That wouldn't allow you to use Taskvars.yml, though.

[andreynering]

Ping @jamestenglish

Does the proposed solution work for you?

I think it's actually cleaner than what you proposed. Thoughts?

[jamestenglish]

@andreynering not really, I saw this was another issue that was closed but I really think your variable precedence scheme is less than ideal. With the current precedence scheme there is no real way to override things when you call a taskfile. Environment variables have the lowest precedence and passing variables through the command line don’t carry through to sub tasks. That makes it really hard to create tasks that have default values that can be overriden and reused.

For example we have one task (broken up into sub tasks) that we want to use in a bunch of different pipelines. The pipelines are very similar so we want a common set of defaults but each pipeline needs to override bits. As far as I can tell my options are:
A) put everything in the top level task so I can override via the cli. This destroys readability.
B) I can use something like taskvars file but again because of the variable precedence it means pipelines have to do file manipulation with something like sed to override. Also I can’t use those as environment values, which spawned my original issue
C) do something kludgy in the task to be able to override things.

Currently I am doing option C but it is making me rethink if go-task is really suitable for generic style tasks. I like the syntax but might go back to make

[smyrman]

With the current precedence scheme there is no real way to override things when you call a taskfile.

There is, but you need to explicitly allow it by getting into a habit of using e.g. the default template function:

version: "2.2"
tasks:
  mytask:
    vars:
      A: '{{.A | default "a"}}'
      B: '{{.B | default "b"}}'
    cmds:
    - echo {{.A}} {{.B}}

$ task mytask A=Hello B=World
echo Hello World
Hello World

Same goes for reusing helper tasks from a pipeline within task; as long as you are into a habit of using the default keyword, you can construct good reusable tasks with sane defaults. You can also prevent values from being (accidentally) overriden, and that is in my view a good thing.

[jamestenglish]

@smyrman as I mentioned in point A that only works for the top level task, sub tasks can’t inherit it. So if you need to add another parameter you have to then pass it through to all your sub tasks

You can’t do:

tasks:
  entry-point:
    vars:
      A: ‘{{.A | default ‘a’}}’
    cmds:
      - task sub-task
  sub-task:
    cmds:
      - echo {{.A}}

(Excuse any formatting I’m on my phone)

Unless you to pass thru every single variable to every subtask

[smyrman]

Indeed, you would need to pass through every single variable, just like you would when calling a function; it would be pretty upsetting if a function inherited local vars that you did not pass in.

version: "2"
tasks:
  entry-point:
    vars:
      A: '{{.A | default "a"}}'
    cmds:
    - task: sub-task
      vars:
        A: '{{.A}}'
  sub-task:
    cmds:
    - echo {{.A}}

$ task A=b
echo b
b

If you need it to be global, it's settable via the environment, but not as input parameters:

version: "2"
vars:
  A: '{{.A | default "a"}}'
tasks:
  entry-point:
    cmds:
    - task: sub-task
  sub-task:
    cmds:
    - echo {{.A}}

$ task entry-point A=b
echo a
a

A=b task entry-point
echo b
b

[andreynering]

Hey @jamestenglish,

It's intentional that the variable precedence order is as is. It's from top to lower level, and as @smyrman, this kinda mimic how function calls would work.

That said, I agree that we could have a way override variables, but I'd prefer that way to be explicit. I'm open to suggestion on the UX perspective (should this be a flag on something?).

---

In the meantime, there's something that could maybe solve your problem today.

This:

env FOO=bar my:task

Is a bit different than this:

task my:task FOO=bar

The former won't override anything, but the latter sets the variable with an equivalent priority to call, which the second in the list of priorities (you can check the priority list here). That mean that only task level variables have a higher priority than that.

I definitely need to improve documentation on some areas. I know some stuff like this is not obvious.

[jamestenglish]

@smyrman

Indeed, you would need to pass through every single variable, just like you would when calling a function; it would be pretty upsetting if a function inherited local vars that you did not pass in.

I disagree, I don’t want a Turing complete, strictly parameterized thing, I want a simple task runner. The fact that go-task already has global variables and a separate file that can act as a global variable set, in my opinion, really moots your argument. Functions already inherit these global variables without having to be passed in.

@andreynering the problem is there is no way to override taskvars or taskfile vars, unless you you do (in my opinion kludgey) something like was suggested where you set defaults for everything and pass in overrides via environment variables. The reason that is kludgey is because environment variables are lowest in the priority list, so you are stuck with either having to add boilerplate so everything is overridable which is not ideal when you want to use the variable priority OR you have values that you can never override. That’s a Sophie’s choice, I want to both be able to keep environment variables at the lowest priority and be able to override global task variables without having to edit the file.

In my opinions this, or a syntax similar to this, should be able to override everything not just call variables to the task

task my:task FOO=bar

When you call the task you should be able to override any type of variable - taskfile var, taskvar, etc. User input at the CLI should always have the highest priority, including global vars. I want environment variables to be environment variables and global vars to be global vars, but I want everything to have the ability to be a game time decision

[andreynering]

Hi again @jamestenglish,

So... some people have started to ask for some stuff that requires minor/small breaking changes to Task. This variable priority thing is one of them. That said, I think now it's a good time to start planning the version 3. I plan to open a meta issue to v3 soon.

The above said, I think I agree that arguments given via CLI should overwrite everything.

I'm not sure about environment variables though, because, for example, the PATH environment variable would disallow any variable to be called PATH inside a Taskfile, which could be confusing to users. Beside this, I'd like to keep the same behavior for both for consistency.

@jamestenglish Do you think that's enough? Anymore more you'd like to propose?

[jamestenglish]

@andreynering first off thanks for being so responsive and engaging, maintaining an OSS project is often a thankless task.

Re other proposals - thinking about it more through this conversation I think the root desire is a mechanism to not make variables need to be explicitly referenced. What I mean by that is this feature request dealt with taking the variables and dumping them into the environment but if there was a mechanism for collecting and looping over variables I could put them into the environment or create a template to pass them into the helm cli myself. Related is issue #140 and #167. Right now the task and the data for that task are pretty tightly coupled - if I add a global variable or a taskvar I have to also update the task to use it. Some tasks that is fine, but there is a category of tasks that you’d like to be able to add data without having to also update the task. Some mechanism to programmatically access variables (either by supporting maps & arrays or through some other thing) would allow utilizing go-task for this other category of problems.

[andreynering]

@jamestenglish Yep, I think #140 is the right way to solve it, together with maybe #82

[davinkevin]

Big +1 on this one!

Especially because we can't provide envs to deps, so we have to use vars.
The main reason why I need this is because tool I use are env-var based in their configuration.

So, I have this kind of things all the time:

  skaffold:dev:
    dir: ~/Workspace/gitlab.com/davinkevin/Podcast-Server
    vars:
      DATABASE_PASSWORD: nAAdo5wNs7WEF1UxUobpJDfS9Si62PHa
      DATABASE_USERNAME: podcast-server-user
      DATABASE_NAME: podcast-server-code-generation
      DATABASE_PORT:
        sh: sh -c "echo \$RANDOM"
    env:
      SKAFFOLD: true
      DATABASE_PASSWORD: '{{.DATABASE_PASSWORD}}'
      DATABASE_USERNAME: '{{.DATABASE_USERNAME}}'
      DATABASE_NAME: '{{.DATABASE_NAME}}'
      DATABASE_PORT: '{{.DATABASE_PORT}}'
    deps:
      - task: k3d:cluster:create
      - task: docker:pg:run
        vars:
          DATABASE_PASSWORD: '{{.DATABASE_PASSWORD}}'
          DATABASE_USERNAME: '{{.DATABASE_USERNAME}}'
          DATABASE_NAME: '{{.DATABASE_NAME}}'
          DATABASE_PORT: '{{.DATABASE_PORT}}'

Which feels a bit strange, not optimum.

/cc @skurtzemann

[JonZeolla]

This is also a common issue with setting TF_VAR_ environment variables for terraform/opentofu

[ccxuy]

Same here lots of variables are passing over again and again, which makes it hard to use task on larger project