Thought experiment: API-centric DX

[shykes]

A thought experiment. What if:

• cloak is purely an API server, packaged as a low-level plumbing tool. No client functionality; no config file; only the bare minimum command-line flags to setup the server, eg. listening address, loglevel.. All configuration is done via the graphql API itself, including loading API extensions (see other points)

• cloakctl could be a companion low-level client tool, similar to buildctl. Since everything is done via graphql, it's really a glorified curl with some conveniences baked in.

• dagger would add a porcelain layer that instruments cloak and adds a nice UX and DX. For example the concept of a project, and corresponding config file; any sort of versioning and package management; universe of extensions; SDKs; a complete CLI experience; interactive development sandbox; etc.

• Because our engine would purely be an API server, our DX would be API-centric. A central concept would be API extensions, which as the name implies, extend the Dagger API with custom features: actions, artifact streams, services, etc. Under the hood this would be powered by cloak's API extension feature.

• Cloak API extensions would be loaded by invoking the special method FS { LoadExtension { Install } }. Or something like that... Basically an extension could be loaded from any FS that complies with the extension ABI.

• To develop a cloak API extension, one must produce a FS that implements the extension ABI. The extension ABI is a spec that describes what cloak needs to run an extension. The spec is pretty simple and defines things like:
  - A valid graphql schema must be present at /cloak/schema.sql
  - the graphql schema must match the following schema: ...
  - a runnable entrypoint must be present at /cloak/run-extension
  - entrypoint will be executed with the root of the extension bundle as its rootfs
  - the following env variables will be set: ....
  - input data will be written to stdin in json
  - output data must be written to stdout in json
  - exit code 0 indicates success. If exit code is an error, errors should be written to stderr in json event stream format. The following json fields have special meaning: ...
  - etc.

• Importantly, Cloak does not define how to build an extension. That is a higher-level concern. Client is responsible for procuring extensions (by building them or downloading them), then loading them from any FS in cloak.

• Dagger adds a whole layer of SDKs, code generation and build tools to make it very easy to create these extensions

• Possibly, even the current project would be built and run in the same way: from cloak's point of view, it would just be another API extension... Except it would be built from the current local directory, and loaded under todoapp instead of netlify or yarn. Makes no difference to cloak.

• I like the idea of telling devops engineers: "you are developing an API". This reminds me of Airtable, where each database has its own API endpoint and generated API docs. So you're not so much using "the Airtable API" as "my food recipe database API powered by Airtable". I think we could copy this concept here. You're not using the Dagger API, you're using the Acme Corp devops API, powered by Dagger. And here are the API docs and generated clients for it.

--> From a technical point of view I suspect this will feel almost identical to what you guys are already seeing and how you see the code working. But at the same time it paints an end-to-end picture that I wasn't seeing before, and that I find very exciting because I see how we can explain it, and what the user experience might look like.

graph BT;
  dagger["dagger CLI"]
  dagger ---> cloak
  cloakctl ---> cloak
  curl ---> cloak
  

  config(["dagger.yaml"])
  config ---> dagger

Loading

[sipsma]

Yes this is a great summary of the general technical direction Andrea and I have been talking about (and that I've been starting to document but then getting caught up in implementation stuff the last few days).

One interesting distinction you make here though is that cloak is different than dagger; cloak is the "kernel" and dagger is like a high-level tool for authoring and loading drivers for that kernel? I was originally thinking cloak will just become dagger, but I like this framing too.

the special method FS { LoadExtension { Install } }

We currently call this import and use the terminology "package":

cloak/api/schema/schema.graphql

Line 42 in cecd88e

import(name: String!, fs: FS): Package

The appeal of "import" and "package" to me is that it cues the equivalent concepts from programming languages and dependency management, but that also may be a product of my own biases. Replacing "package" with "extension" or "plugin" seems entirely reasonable too.

To develop a cloak API extension, one must produce a FS that implements the extension ABI. The extension ABI is a spec that describes what cloak needs to run an extension.

Yes you can see the current (very rough, informally defined) version of this here:

• Cloak-side
• Action-side:
  • Go
  • TS

But now that the switch to the new resolver model is done (which impacted this a lot), it's definitely time to start formally defining this (and cleaning up the current implementation, which is a bit hacky in places). That will make it much easier to start adding new runtimes more quickly.

Overall, were definitely on the same page here. We should discuss more and then I'll incorporate this into the docs/architecture.md I've been working on so which can be a first draft of our "spec".

[shykes]

Yes this is a great summary of the general technical direction Andrea and I have been talking about (and that I've been starting to document but then getting caught up in implementation stuff the last few days).

OK good to know we're still aligned on the fundamentals :) A good time to start digging into the next level of detail.

One interesting distinction you make here though is that cloak is different than dagger; cloak is the "kernel" and dagger is like a high-level tool for authoring and loading drivers for that kernel? I was originally thinking cloak will just become dagger, but I like this framing too.

Yes, it's not the only possible approach, but I find it promising. Some of the benefits I see:

• It makes for better "technical marketing": easier to explain how it relates to our current stack, and in particular porting dagger 0.2 on top of cloak; easier to explain as an exciting new thing rather than a big breaking change to the existing thing.
• It provides an answer to the problem of embedding dagger in non-Go languages. Since it's not practical to re-implement cloak itself in several languages (which would pull in llb implementation, buildkit grpc client implementation etc); then our engine will be a runtime dependency for all programs that embed it. Ideally this runtime dependency would be a small binary with a stable interface and feature set; that can fork-executed with minimal configuration and state, and instrumented over an API. A stripped-down, stable cloak fits that description better than a full-featured, fast-moving dagger.
• It creates a distinct community tier. A standalone cloak will attract a different kind of open-source community than the complete dagger. This is because Dagger is a complete product, whereas Cloak is a component. Dagger will attract a community of power users of our product; whereas cloak will attract a community of people building their own products (possibly competitive with us) that want to collaborate on the low-level tech. This is similar to the buildkit, containerd, runc communities as opposed to the Docker community. This reduces drama, and maximizes the size and throughput of each community. Everybody wins.
• Since cloak is (obviously) a crucial component of dagger, if the only way to instrument it is via the graphql API, then it puts maximum pressure on the graphql API to be feature-complete, stable and documented. No cheating... which will help us have a clean architecture. Example: how do we stream local directory? how do we export artifacts? how do we stream logs? how do we proxy sockets? interactive tty to the container for debugging? The graphql API will need to cover all of this. It's essentially a buildkitd wrapper.
• Separating concerns early, will help us experiment faster on the high-level DX and UX. Many people can target the cloak API in parallel, and develop custom clients without fearing that they will collide with the core cloak development effort. This will accelerate the design process and ultimately produce a better DX.

the special method FS { LoadExtension { Install } }

We currently call this import and use the terminology "package":

[...]

The appeal of "import" and "package" to me is that it cues the equivalent concepts from programming languages and dependency management, but that also may be a product of my own biases. Replacing "package" with "extension" or "plugin" seems entirely reasonable too.

• Yes that terminology "shift" is a big aspect of this thought experiment of an API-centric DX. The bet is that we change the framing to developers, away from "you are writing a better script, and this runtime will execute it", and towards "you are creating an automation API, and this engine will serve it". I'd like to explore the ramifications of this change of framing.

cloak/api/schema/schema.graphql

Line 42 in cecd88e

import(name: String!, fs: FS): Package

A note on the GraphQL schema, ignoring terminology: wouldn't it make more sense to make Import/LoadExtension a method of FS rather than a top-level query? This would allow chaining it at the end of the complete query required to build or download the FS.

eg.

Query LoadNetlify {
  core {
    localdir(".") { 
      dockerbuild(dockerfile: "./examples/netlify/Dockerfile") {
        fs {
          LoadExtension {
              install
              installDev: install(name: "netlify-dev", debug: true)
          }
        }
      }
    }
  }
}

I find that pattern super powerful. I even imagine that you could pass individual graphql files to cloak at start, as initialization scripts. Or even a whole directory of them, to be interpreted in alphabetical order. The successor of /etc/init.d :)

To develop a cloak API extension, one must produce a FS that implements the extension ABI. The extension ABI is a spec that describes what cloak needs to run an extension.

Yes you can see the current (very rough, informally defined) version of this here:

• Cloak-side

• Action-side:

  • Go
  • TS

Awesome. The key is to decouple the build and run phase, by stabilizing the spec of the runnable bundle. This is what allows the separation of concerns between dagger and cloak, as well as easier innovation and experimentation at the DX layer.

The key though is that, whatever tooling you use to build or download extensions, it's all cloak instructions in the end. Hence the importance of the chaining part.

Another super important aspect is that all instrumentation of cloak happens over its API. No state at all if possible; no configuration files; no standard ports or unix sockets paths; just a stateless executable binary that you run, then configure and instrument purely by graphql calls.

[sipsma]

First attempt at documenting cloak concepts (and implicitly the architecture) which begins to incorporate some of the ideas from this discussion: https://github.com/dagger/cloak/blob/e37ac84ea39af13d1c45f5923eef339edbf5903b/docs/concepts.md

I said before in this discussion that what you described here was the direction we were going, but then as I wrote the doc and thought about it more I realized that "full" api-centricity was more than I was originally imagining. Taken to its logical extreme there are lots of things I previously assumed would be baked into cloak that actually could be modeled as extensions.

For example, in the doc I wrote stubbers are extensions, SDKs are actually just extensions which can create new extensions, etc. But the design in my head (and sort of implied in the doc) currently still hardcodes the "runtime protocol" into cloak and assumes all resolvers "speak it" even though that too could probably be an extension in theory.

I agree with Andrea's point made in separate discussions that taking the api-centricity to the absolute extreme where cloak becomes as utterly minimal as possible may impose significant some significant cognitive+technical complexity. It's often just easier to understand architecture w/ "baked-in" functionality than it is to understand one where everything is a pluggable extension. I tried taking it to the extreme when writing the doc but my brain's stack was overflowing and I had to scale back.

Overall, on a product level we can absolutely still decide to take this api-centric approach where cloak and dagger are cleanly separated, I think we on a technical level just need to move slowly towards it, migrating functionality out of cloak to dagger piecemeal. This approach also lets us assess it on a product level too I suppose; if we get feedback from users that they want more customization of components like the runtime protocol, that will be a good indication that full api-centricity is worth pursuing.

[shykes]

I would need to discuss live what you have in mind with the “utterly minimal” part and its cognitive cost. It makes sense in theory but depends on what features we’re talking about.

What I think is important architecture-wise, is to avoid a spaghetti situation, where the boundaries of the API are unclear and partially implicit. I think defining a crystal-clear API boundary now, making explicit what is inside and outside, and then enforcing it strictly, is crucially important if we are serious about building a platform. This cannot be bolted on later. It’s different from product packaging (what goes into a binary called cloak vs dagger how do we distribute these things) and technical implementation (how does the graphql resolver work, do we use federation or not). It’s even different from DX polish (do we use yaml in config files; what are the cli flags). All those things can change over time without too much trouble. But this core architectural decision of API boundary will not. It’s more than merely implementation detail.

[shykes]

But the design in my head (and sort of implied in the doc) currently still hardcodes the "runtime protocol" into cloak and assumes all resolvers "speak it" even though that too could probably be an extension in theory.

Could you elaborate on this part? I didn't understand it.

[sipsma]

Could you elaborate on this part? I didn't understand it.

Yeah so right now the cloak server is hardcoded with the runtime protocol (here): if you load an extension you must provide a Filesystem containing the resolvers for that extension and that Filesystem must implement the "handler-side" of the runtime protocol. This involves hardcoding into cloak a lot of aspects of that protocol (how inputs are provided, how outputs are obtained, the location of the socket back to cloak, etc.)

In theory, I think it would actually be possible for that to not be hardcoded in cloak. The whole protocol is essentially just the creation of an ExecOp, so in theory you could represent this by just re-using core's Exec. In practice, this would probably mean that instead of an extension requiring a Filesystem be provided, you'd provide a chained graphql query that should be run to obtain the output of the resolver.

There's appeal in this due to the flexibility it grants and the simplicity it provides to the server. But downsides include that A) complexity is pushed to the client (though that could probably be alleviated by layering on more extensions that bootstrap the logic clients need) and B) this gets so abstract and complicated that my brain starts hurting :-)

Not sure if this helps or hurts, agree that a live discussion will help at this point.

All those things can change over time without too much trouble. But this core architectural decision of API boundary will not.

I agree completely (have to go get ready for the next demo so can't finish thoughts now, but can discuss more)

[shykes]

Yeah so right now the cloak server is hardcoded with the runtime protocol (here): if you load an extension you must provide a Filesystem containing the resolvers for that extension and that Filesystem must implement the "handler-side" of the runtime protocol. This involves hardcoding into cloak a lot of aspects of that protocol (how inputs are provided, how outputs are obtained, the location of the socket back to cloak, etc.)

In theory, I think it would actually be possible for that to not be hardcoded in cloak. The whole protocol is essentially just the creation of an ExecOp, so in theory you could represent this by just re-using core's Exec. In practice, this would probably mean that instead of an extension requiring a Filesystem be provided, you'd provide a chained graphql query that should be run to obtain the output of the resolver.

There's appeal in this due to the flexibility it grants and the simplicity it provides to the server. But downsides include that A) complexity is pushed to the client (though that could probably be alleviated by layering on more extensions that bootstrap the logic clients need) and B) this gets so abstract and complicated that my brain starts hurting :-)

OK, I understand now. Yes, this would be a case of too much minimalism in the core. IMO the extension system should be part of core. There should only be one way to extend the API, otherwise we risk fragmenting the ecosystem of extensions. It's important that any Dagger user knows how to load any extension in any engine.