Federation Implementation

[calvinrp]

Federation is one of the few outstanding protocol features not implemented.

Effort is underway to update the docs with targeted use cases. To highlight two use cases:

• Registry packages that import packages published to other registries;
• Registry mirrors;

This proposed implementation can be thought of as a composition of three new features:

• Proxying on behalf of another registry;
• Mapping namespace to published registry;
• Aliasing registry namespaces;

Registry Proxying

Simply adds a request header (perhaps named Warg-Registry) that specifies the registry domain name that is the registry subject of the API request. If that header is present, the registry server would either respond like a read-thru cache of the other registry's API or error if that registry is not supported. A registry could choose to support (or not) passing along publish requests to the subject registry.

Since this is just an HTTP request header, there are many implementation options. For instance, a proxy service could sit in front of the registry servers and route the request based upon the header.

Registries are expected to be selective which other registries (if any) that they support proxying.

Namespace Mapping

A package identifier, such as my-company:foo-package, does not indicate the registry domain name where it is published. Also, namespaces (my-company in this example) are not expected to be globally unique across all registries.

The client CLI needs to know which registry has which package. Also, when publishing packages, a registry needs to validate package imports and know which registry has which package.

This implementation proposes using namespaces to group package identifiers and associate them with their published registry domain name.

This proposes a new registry API endpoint for the client to retrieve the registry's namespace mappings to its published registry domain names. This is the mappings that the registry will use to validate package imports published to the registry. The registry will also indicate if it supports proxying those registry domain names.

The client CLI can use the mappings provided by the registry as well as extend them or override them. This configuration can be set for user profile global use or project-based with a lock file. The lock file includes the checkpoints for each registry.

Namespace Aliasing

The following proposes an extension to the namespace mapping configuration.

To resolve naming conflicts, namespace mappings can alias the source registry's namespace to a new name to be used by packages published to its registry.

By default, package imports are resolved in the context of the mappings of the registry it was published. To resolve package imports as if it was published to a different registry, the namespace mappings can specify a registry context.

[lann]

perhaps named X-Registry

The x- prefix is no longer recommended. Maybe Warg-Registry?

[calvinrp]

Right. Old habits die hard. Works for me.

[lann]

I'm still very fuzzy on how namespace mapping/aliasing would work in practice.

If I publish a package with a dependency on some-namspaced:package with the intention that it points at a specific implementation, who can override that intention? It makes sense that the end-developer would be able to, but what about a registry?

I can imagine scenarios where the registry I published to would be able to replace the dependency (via a "trust reset" of the package name); I have already trusted the registry to some extent to manage my package name so that doesn't seem like a major leap.

Allowing a proxying registry to override a dependency is harder to reason about; certainly there are use-cases for it (my company needs to swap in a dependency hotfix) but I'm not sure how to balance that against a potential for misuse.

This may ultimately be more of a question of UX than technical features, but I think in this kind of design spec'ing the UX can be as important as protocol design.

[lukewagner]

In my framing above, there are two distinct things you can do: "import" a namespace (from one registry into another registry), and "override" (a package in an imported namespace). Given this framing, I don't see where the need to "override" a namespace would arise, but I think perhaps it addresses the use cases you're imagining?

Also, just to paint a richer picture: I could imagine there being multiple distinct override operations (such that an "override" was like an instruction that selected one of these operations and supplied immediate operands): overriding a specific version (effectively just swapping out the contents); overriding a whole package (replacing the whole package log); hiding a specific package (as if it wasn't in the namespace); hiding all packages that are not on a given list.

[esoterra]

These overrides you're describing are a bit different from the aliases I was proposing and that may be part of why we disagreed on terminology yesterday. I use the term alias to describe the operation "import namespace <A> as <B>", which is useful for example if you want to include packages from multiple other registries but they define overlapping namespaces.

I've so far been hoping to avoid getting too deep into per-package overrides/mappings (until we're a little further in) because the volume of information required to represent this will be quite large, embedding it efficiently in the data structures will be complicated, and not embedding it in the data structures opens up some security concerns.

[calvinrp]

Overrides, as described here, feel like they could be added as a feature later on. Or do you think it needs to be tackled at the start?

Conceptually, something like this might get us most of the way there:

import { "b-namespace" as "name-for-b-in-this-registry" } from "registry.b.com"
import { "c-namespace" } from "registry.c.com"

[lukewagner]

@esoterra Ah hah, that helps explain alot. Yes, "alias" in your original meaning makes more sense and is quite distinct from "overrides". I think "import" is the closest match to module terminology (as all 3 of our metaphors show), so maybe we could all use that term going forward? (In theory, we could add an "alias" feature to registries in the future that was exactly like component-model aliases: it would give you the ability to take an existing package and re-publish it (its content and log) with a second name (in the same or different namespace), which you could imagine being useful in certain esoteric situations.)

@calvinrp I agree that overrides (at least as I described them above) could be added later on; I was mostly just sketching them out to help build a shared understanding of what this "override" is and how it relates to a near-er term "import" feature that seems like the foundation of federation.

[calvinrp]

"Import namespace", naming convention works for me. @lukewagner Thanks for clarifying and thinking through.

[calvinrp]

I was proposing Namespace Mapping and Aliasing. Do we need finer grain mapping/aliasing down to the Package Identifier?

[esoterra]

I think I agree with Lann in the sense that a client can choose where/how to lookup packages however they want including at the individual package level. I don't however agree with Calvin in that I think it's much much more practical/useful/obvious for registries to map things at the namespace level and produce the general property that things from the namespace come from the same origin.

i.e. I think the protocol should concern itself only with defining mappings from namespaces to origins not from packages.

[calvinrp]

Thinking through...

If a registry wanted to fork a specific package dependency from another registry, it would remove that namespace alias so that it would resolve to packages in its registry. You'd publish (or publish copies) the packages (and package deps) that you need.

Alternatively, the registry could change the namespace alias to another registry that has the forked packages. That might be cleaner.

[esoterra]

Ya, I think it'd be better to have a registry decide it's going to fork packages from some namespace and use one of the other techniques (e.g. log copying, log rewriting) on the whole of that namespace. It's just going to get incredibly messy if registries have a bunch of package-specific origin mappings and then you try to include that registry in another registry with package-specific mappings...

I think namespaces are a much more practical unit to map with.

[calvinrp]

I'll let others chime in. But I'm OK with that.

[lukewagner]

I think the sketch I put above ends up including this as a basic part of the design?

[calvinrp]

Proposed API Changes:

Warg-Registry HTTP Header

For all API endpoints, if the Warg-Registry header is present in the request, the response will be as if the registry at the domain name specified in the header value was the one responding.

The response will include the same Warg-Registry header and value if the server is responding as that registry. If the response header is omitted, the client should assume that the server ignored the request header.

GET /v1/namespace/imports

Response is a JSON:

    {
        "{namespace}": {
            "namespace": "{source-registry-namespace}",
            "registry": "{registry-domain-name}",
            "direct": true
        }
    }

For each imported namespace, there is an object key that is the imported namespace name with an object that specifies the source registry's namespace (omitted, if imported as the same name) and the registry domain name.

During package publishing, the registry will validate package imports against what is available in its own registry and these provided imported namespaces from other registries.

If the registry does not support using the Warg-Registry subject header for specified registry domain name, the direct field is set to true.

POST /v1/package/{logId}/record (Modify Current Endpoint)

Extend the current publish package record endpoint with an optional field namespaceImports in the request body.

When publishing a package with an import of another registry with a namespace not currently present in the registry namespace imports, the client will include the namespaceImports as part of the publish request.

For each missing imported namespace, there is an object key that is the imported namespace name with an object that specifies the source registry's namespace (omitted, if imported as the same name) and the registry domain name.

The registry server can choose to adopt policies on whether to accept the publish or reject. If a registry does not allow importing new namespaces through this API, it will always reject publishes with unknown imports.

After publishing, the newly imported namespaces are expected to be present in the GET /v1/namespace/imports endpoint.

Example:

{
  "packageId": "fun-corp:most-fun-pkg",
  "record": {
    "contentBytes": "ZXhhbXBsZQ==",
    "keyId": "sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c",
    "signature": "ecdsa-p256:MEUCIQCzWZBW6ux9LecP66Y+hjmLZTP/hZVz7puzlPTXcRT2wwIgQZO7nxP0nugtw18MwHZ26ROFWcJmgCtKOguK031Y1D0="
  },
  "namespaceImports": {
     "wasi": {
         "registry": "registry.wasi.dev"
     },
     "ba": {
         "registry": "registry.bytecodealliance.org",
         "namespace": "bytecode-alliance"
     }
  }
}