wit/gen: scaffolding for generating a series of Go packages and files

wit: ID type

This will be removed in a future commit.

wit/gen: add Package.Decls

internal/go/gen, bindgen, wit: refactor

bindgen: Option types

bindgen: initial commit of func Go()

cmd/wit-bindgen-go: syntax -> wit

cmd/wit-bindgen-go: add basic generate command

internal/go/gen: Ident type

bindgen, internal/go, wit: WIP

Generating valid Go names from WIT names:
1. Scan for an explicit mapping, e.g. wasi:clocks/wall-clock -> wasi/clocks/wallclock
2. Perform string conversion, e.g. : -> /, strip -
3. (optional) validate short package name as valid Go identifier
4. Check against list of Go reserved words
5. For decl names, convert to Go-style names (TitleCase), check against existing decl

bindgen, internal/go: WIP

cmd/wit-bindgen-go: add --map wit=go ... flag

bindgen, cmd/wit-bindgen-go: implement generated by for DO NOT EDIT comment

internal/design: stab at what a generated interface for wasi:clocks/wall-clock could look like

_design: multiple Go modules

_design: WIP for imported and exported type

_design/import-and-export: add README

_design/import-and-export: fix typo

bindgen, wit: WIP

remove go.work and secondary go.mod file

_design -> design, with go:build tags

wasm/cm: importable Component Model primitive types

design/wasi/io/poll: hand-rolled example for wasi:io/poll

design/wasi/io/poll: right ABI?

design/import-and-export: remove build tags where not necessary

design/wasi/io: update WIP

design/wasi/io/poll: improve comments

design/wasi/io/poll: wasmexport methods should call the real methods on the type

design/wasi/io: add world.wit source

design: reorganized

design/{export,import}: further explorations

design/export/wasi/io/poll: exploring own/borrow semantics

wasm/cm: Resource, Handle, Own, Borrow types

wasm/cm: Resource, Handle, Own, Borrow types

design: further experimentation

cabi, design: post-return function to free allocated memory

wasm/cabi: combine cabi and wasm/cm packages

wasm/cabi: update README links

cmd/wit-bindgen-go/cmd/generate: update to github.com/urfave/cli/v3 API changes

cabi, design: move cabi back out of wasm directory

cabi: add some TODOs

wit: rename PackageName to ID; add extension (world, interface)

bindgen, wit: update use of ID

wit, testdata: ID -> Ident

internal/go/gen: fix test

wit: move Ident to ident.go and add tests

bidngen, internal/go/gen: start generating Go names

bindgen, internal/go/gen: more Go code generation; WIP

cmd/wit-bindgen-go/cmd/generate, internal/go/gen: PackagePath function to load go.mod file

cmd/wit-bindgen-go/cmd/generate: add -o flag

internal/go/gen: Unique name generator

internal/go/gen: add package, file, import manipulation

bindgen, cmd/wit-bindgen-go/cmd/generate: PackageRoot option

bindgen, wit: generate multiple Go packages

design/wasi/io/streams: try version directories

bindgen: WIP on package path and name generation

bindgen: update comments

bindgen: WIP

bindgen, cmd/wit-bindgen-go/cmd/generate: remove - from Go package paths

wit: fix typo

bindgen: WIP emitting Go names for types in correct packages

bindgen: emit functions; WIP

bindgen: remove packageForWorld method

bindgen: fake wasmimport lines

bindgen: add TODO about mapping WIT functions and typedefs to Go decls

internal/go/gen: replace Imports type with Imports func

bindgen: export Go name helpers

bindgen: rename isSep to notLetterDigit

internal/go/gen: rename AddImport to Import

bindgen, internal/go/gen: update Unique function

internal/go/gen: NewFile -> File

bindgen, internal/go/gen: update AddDecl

bindgen, internal/go/gen: Decl returns Decl{}

bindgen: map wit.TypeDef and wit.Function to gen.Decl

bindgen: update comments

cmd/wit-bindgen-go/cmd/generate: note generated package list

internal/go/gen: remove File.ImportDecls

internal/go/gen: FormatDocComments

bindgen, cmd/wit-bindgen-go/cmd/generate, internal/go/gen: generate Go code into individual files in packages

Write doc comments from WIT into files; set type aliases to any // TODO

bindgen: serialize WIT docs into Go doc comments for types and functions

bindgen: start breaking out type generation

bindgen: rearrange deck chairs

bindgen: stub out option<T>

internal/go/gen: use golang.org/x/mod/modfile

internal/go/gen: require IsGo for HasPackageDocs

internal/go/gen: reorganize

wit: omit doc comments when ctx == nil

bindgen: print WIT definition in doc comments for each type

wit: use tab indentation instead of 4 spaces

This is necessary for Go doc comments that embed original WIT definition.

testdata: regenerate with tab indentation

bindgen: declare types before defining them

wit: test that all TypeDef roots have named owners

Anonymous interfaces can use types, but should point to a type owned by a named interface.

design/wasi/io/poll/exports: update design for exported interfaces

cabi: tweak

cabi/README.md

@@ -1,6 +1,12 @@
-### Usage
+# Canonical ABI
 
-The `cabi` package contains a single exported WebAssembly function `cabi_realloc` ([Canonical ABI] realloc). To use, import this package with `_`:
+## Types
+
+Package `cabi` declares a number of types, including [Component Model](https://component-model.bytecodealliance.org/) [primitive types](https://component-model.bytecodealliance.org/design/wit.html#primitive-types), along with resource and handle types.
+
+## cabi_realloc
+
+The `cabi` package contains an exported WebAssembly function `cabi_realloc` ([Canonical ABI] realloc). To use, import this package with `_`:
 
 ```
 import _ "github.com/ydnar/wasm-tools-go/cabi"

cabi/memory.go

@@ -0,0 +1,35 @@
+package cabi
+
+import (
+	"sync"
+	"unsafe"
+)
+
+// TODO: remove this or move it to package cm?
+
+var (
+	mu       sync.Mutex
+	pointers = make(map[unsafe.Pointer]int)
+)
+
+// KeepAlive reference counts a pointer.
+// TODO: prove this works.
+func KeepAlive(ptr unsafe.Pointer) {
+	mu.Lock()
+	n := pointers[ptr]
+	pointers[ptr] = n + 1
+	mu.Unlock()
+}
+
+// Drop drops a reference to ptr.
+// TODO: prove this works.
+func Drop(ptr unsafe.Pointer) {
+	mu.Lock()
+	n := pointers[ptr]
+	n -= 1
+	// TODO: panic if n < 0?
+	if n <= 0 {
+		delete(pointers, ptr)
+	}
+	mu.Unlock()
+}

cabi/primitive.go

@@ -0,0 +1,17 @@
+package cabi
+
+// TODO: remove this or move it to package cm.
+
+type Bool bool
+type S8 int8
+type U8 uint8
+type S16 int16
+type U16 uint16
+type S32 int32
+type U32 uint32
+type S64 int64
+type U64 uint64
+type F32 float32
+type F64 float64
+type Char rune
+type String string

cabi/resource.go

@@ -0,0 +1,36 @@
+package cabi
+
+// TODO: remove this or move it to package cm.
+
+// Resource is the interface implemented by all [resource] types.
+type Resource[T any] interface {
+	ResourceHandle() Handle[T]
+	// BorrowResource() Borrow[T]
+	// OwnResource() Own[T]
+}
+
+// Handle is an opaque handle to a [resource].
+type Handle[T any] uint32
+
+// Own is a handle to an owned [resource].
+type Own[T any] Handle[T]
+
+func (o Own[T]) Rep() T {
+	return Rep(Handle[T](o))
+}
+
+// Borrow is a handle to a borrowed [resource].
+type Borrow[T any] Handle[T]
+
+func (b Borrow[T]) Rep() T {
+	return Rep(Handle[T](b))
+}
+
+// TODO: can we use finalizers for dropping handles?
+
+// Rep returns the representation of handle, if any.
+func Rep[T any, H Handle[T]](handle H) T {
+	// TODO: extract the actual representation from a table
+	var v T
+	return v
+}

design/README.md

@@ -0,0 +1,3 @@
+# Design
+
+This directory contains hand-written Go packages that correspond to WIT definitions for the purposes of fleshing out a viable design. None of these packages should be imported or used.

design/import-and-export/README.md

@@ -0,0 +1,5 @@
+# import-and-export
+
+This design example contains a subset of the `wasi:clocks` package with a world that both *imports* and *exports* the `wasi:clocks/wall-clock` interface. The purpose of this example is to sketch out a mechanism for generating Go code whereby an interface (and types) can simultaneously be imported and exported.
+
+The source WIT file is [world.wit](./world.wit).

design/import-and-export/wasi/clocks/wallclock/exports.go

@@ -0,0 +1,22 @@
+//go:build wasm
+
+package wallclock
+
+// Instance is the sole global instance of the
+// WIT interface "wasi:clocks/wall-clock".
+// Assign it to accept calls to this interface.
+var Instance Interface
+
+// FIXME: correct type for struct return values
+//
+//go:wasmexport wasi:clocks/wall-clock now
+func wasmexport_now() DateTime {
+	return Instance.Now()
+}
+
+// FIXME: correct type for struct return values
+//
+//go:wasmexport wasi:clocks/wall-clock resolution
+func wasmexport_resolution() DateTime {
+	return Instance.Resolution()
+}

design/import-and-export/wasi/clocks/wallclock/imports.go

@@ -0,0 +1,25 @@
+//go:build wasm
+
+package wallclock
+
+// Now returns a DateTime for the current wall clock, corresponding to
+// the Component Model function "wasi:clocks/wall-clock.now".
+func Now() DateTime {
+	return wasmimport_now()
+}
+
+// FIXME: correct type for struct return values
+//
+//go:wasmimport wasi:clocks/wall-clock now
+func wasmimport_now() DateTime
+
+// Resolution returns the resolution of the current wall clock, corresponding to
+// the Component Model function "wasi:clocks/wall-clock.resolution".
+func Resolution() DateTime {
+	return wasmimport_resolution()
+}
+
+// FIXME: correct type for struct return values
+//
+//go:wasmimport wasi:clocks/wall-clock resolution
+func wasmimport_resolution() DateTime

design/import-and-export/wasi/clocks/wallclock/types.go

@@ -0,0 +1,13 @@
+package wallclock
+
+// Interface is the Go implementation of WIT interface "wasi:clocks/wall-clock".
+type Interface interface {
+	Now() DateTime
+	Resolution() DateTime
+}
+
+// DateTime is a Go implementation of WIT type "wasi:clocks/wall-clock.datetime".
+type DateTime struct {
+	Seconds      uint64
+	Nanonseconds uint32
+}

design/import-and-export/world.wit

@@ -0,0 +1,15 @@
+package wasi:clocks;
+
+interface wall-clock {
+    record datetime {
+        seconds: u64,
+        nanoseconds: u32
+    }
+    now: func() -> datetime;
+    resolution: func() -> datetime;
+}
+
+world import-and-export {
+    import wall-clock;
+    export wall-clock;
+}

design/import-and-export/world.wit.json

@@ -0,0 +1,82 @@
+{
+  "worlds": [
+    {
+      "name": "import-and-export",
+      "imports": {
+        "interface-0": {
+          "interface": 0
+        }
+      },
+      "exports": {
+        "interface-0": {
+          "interface": 0
+        }
+      },
+      "package": 0
+    }
+  ],
+  "interfaces": [
+    {
+      "name": "wall-clock",
+      "types": {
+        "datetime": 0
+      },
+      "functions": {
+        "now": {
+          "name": "now",
+          "kind": "freestanding",
+          "params": [],
+          "results": [
+            {
+              "type": 0
+            }
+          ]
+        },
+        "resolution": {
+          "name": "resolution",
+          "kind": "freestanding",
+          "params": [],
+          "results": [
+            {
+              "type": 0
+            }
+          ]
+        }
+      },
+      "package": 0
+    }
+  ],
+  "types": [
+    {
+      "name": "datetime",
+      "kind": {
+        "record": {
+          "fields": [
+            {
+              "name": "seconds",
+              "type": "u64"
+            },
+            {
+              "name": "nanoseconds",
+              "type": "u32"
+            }
+          ]
+        }
+      },
+      "owner": {
+        "interface": 0
+      }
+    }
+  ],
+  "packages": [
+    {
+      "name": "wasi:clocks",
+      "interfaces": {
+        "wall-clock": 0
+      },
+      "worlds": {
+        "import-and-export": 0
+      }
+    }
+  ]
+}

design/wasi/io/poll/exports/exports.go

@@ -0,0 +1,54 @@
+package exports
+
+import (
+	"unsafe"
+
+	"github.com/ydnar/wasm-tools-go/cabi"
+	"github.com/ydnar/wasm-tools-go/cm"
+)
+
+// Interface implements the Component Model interface "wasi:io/poll".
+type Interface interface {
+	Poll(in cm.List[cabi.Borrow[Pollable]]) cm.List[uint32]
+	Pollable() interface {
+		// constructor, static functions would go here
+	}
+}
+
+// Export registers a concrete implementation of "wasi:io/poll".
+func Export(i Interface) {
+	impl = i
+}
+
+// TODO: make a default implementation that panics with a helpful message on all function calls.
+var impl Interface
+
+//go:wasmexport wasi:io/poll#poll
+func poll(in cm.List[cabi.Borrow[Pollable]], result *cm.List[uint32]) {
+	*result = impl.Poll(in)
+	// sData := unsafe.SliceData(s.Slice())
+	// cabi.KeepAlive(unsafe.Pointer(sData))
+}
+
+//go:wasmexport wasi:io/poll#cabi_post_poll
+func cabi_post_poll(result *cm.List[uint32]) {
+	// Is this necessary if the Go GC runs after the wasmexport call?
+	cabi.Drop(unsafe.Pointer(result.Data()))
+}
+
+//go:wasmexport wasi:io/poll#[method]pollable.block
+func method_pollable_block(self cabi.Borrow[Pollable]) {
+	self.Rep().Block()
+}
+
+//go:wasmexport wasi:io/poll#[method]pollable.ready
+func method_pollable_ready(self cabi.Borrow[Pollable]) bool {
+	return self.Rep().Ready()
+}
+
+// Pollable represents the Component Model type "wasi:io/poll.pollable".
+type Pollable interface {
+	Block()
+	Ready() bool
+	cabi.Resource[Pollable]
+}