Proposal: Allowing WIT bindgen written in languages they are generating for

[Mossaka]

@ydnar and I originally posted this little proposal on this zulipchat post and I am raising an issue here for a bigger exposure.

Challenges

Currently, all the WIT binding generators are written in Rust to emit bindings in different languages. Given that the parsed WIT AST (in particular, Resolve struct) is represented in Rust data structures, this makes sense. However, this poses some challenges to language experts to develop and maintain WIT binding generators because they will have to know Rust well. This motivates the idea of developing bindgen in their own languages consuming resolved WIT packages. The question is how and in what format can we supply resolved WIT packages to bindgen written in different languages.

Details

wit-bindgen is a tool to generate guest language bindings for .wit files for languages that compile to Wasm components. It leverages wit-parser crate from wasm-tools to parse and interpret WIT packages. There is a clear difference between wit-bindgen and wit-parser:

1. wit-parser parses the input WIT files, resolves all the imports, including foreign imports, and represent all the worlds and interfaces in a struct called wit_parser::Resolved.
2. wit-bindgen uses fully-resolved WIT packages in wit_parser::Resolved to generate bindings. Roughly speaking, a new language bindgen needs to implement these Rust traits:
   1. wit_bindgen_core::Bindgen - lower and lift signatures from Wasm types to WIT types
   2. wit_bindgen_core::WorldGenerator - generate source code to files for WIT worlds
   3. wit_bindgen_core::InterfaceGenerator - generate source code to files for WIT interfaces

Ideas

1. Complete rewrite of wit-parser and wit-bindgen in each language. For example, wit-parser-go and wit-bindgen-go for WIT binding Go generator. See an example for wit-parser in Go by @jordan-rash
2. Reuse wit-parser as much as possible by serializing its exported modules (mainly wit_parser::Resolve) and deserialized to each binding generators in their respective languages for consuming. @alexcrichton has some insights into this.
3. @lann suggested to turn language generators to Wasm components by defining generator interfaces as WIT packages.

[cpetig]

I suggest to also consider exporting parts of the abi crate (which controls the lifting and lowering mechanisms; currently in move to wit-bindgen - see bytecodealliance/wasm-tools#1149 for details) to other languages as getting the ABI conventions right will be the more thankless part of language bindings generation.

[ydnar]

I think (2) makes sense, and (3) after a language has been bootstrapped.

I suggest to also consider exporting parts of the abi crate (which controls the lifting and lowering mechanisms; currently in move to wit-bindgen - see bytecodealliance/wasm-tools#1149 for details) to other languages as getting the ABI conventions right will be the more thankless part of language bindings generation.

Would it be possible to annotate the Resolve struct (or its JSON representation) with ABI information so target language generators can correctly lift/lower?

[lann]

As an extension to "turn language generators to Wasm components", if the generator interfaces can avoid resources [edit: actually I'm not sure this is even necessary] then bootstrapping could also be done by adapting the interfaces to some RPC system with widespread implementations like gRPC or JSON-RPC. This RPC adapter could itself be generated by the same system.

[ydnar]

Makes sense. Question—what would require bidirectional communication between wit-bindgen and, say, wit-bindgen-go? Versus, say, just piping JSON into another process?

I’m probably missing something! (asking because an RPC layer, even if autogenerated, is another order of complexity versus just parsing some intermediate representation)

[lann]

what would require bidirectional communication

It depends on how much wit-bindgen code you want to share with generators. With no sharing, this approach would be no better than approach 2 (serializing Resolve), but that leaves a lot of common functionality up to every bindings generator as alluded to in #640 (comment).

[ydnar]

what would require bidirectional communication

It depends on how much wit-bindgen code you want to share with generators. With no sharing, this approach would be no better than approach 2 (serializing Resolve), but that leaves a lot of common functionality up to every bindings generator as alluded to in #640 (comment).

Repeating question from above, in case it got lost: Would it be possible to annotate the Resolve struct (or its JSON representation) with ABI information so target language generators can correctly lift/lower?

[lann]

Would it be possible to annotate the Resolve struct (or its JSON representation) with ABI information so target language generators can correctly lift/lower?

I am not familiar enough with that code to be confident, but I suspect the answer is "possible but somewhat difficult, especially to maintain".

[ydnar]

Got it. How would you envision an RPC flow between wit-bindgen and a language specific tool? Specifically for ABI related code gen?

[lann]

Off the cuff, I might expect a "generator component" to have exports corresponding to e.g. WorldGenerator and Bindgen and imports corresponding to portions of what is currently in the public interface of wit-parser, e.g. some methods of Resolve.

[Mossaka]

The Pr to add Serde::Serialize to wit-parser has been merged! We are now able to consume the JSON output to write Go binding generator in Go, where @ydnar is making some progress at https://github.com/ydnar/wasm-tools-go

[jordan-rash]

The Pr to add Serde::Serialize to wit-parser has been merged! We are now able to consume the JSON output to write Go binding generator in Go, where @ydnar is making some progress at https://github.com/ydnar/wasm-tools-go

Amazing! Thanks everyone that worked to get this merged!!

[ydnar]

I’ve introduced some aspects of the Component Model ABI into wasm-tools-go, namely Type, TypeDef, and TypeDefKind conform to an ABI interface with (currently) two methods:

1. Size() uintptr
2. Align() uintptr

Docs here, which link to the canonical ABI documentation that this is based on. I’d love some feedback on this direction.

[ydnar]

To help me understand the WIT spec and the data structures from the wit-parser crate, I have a branch that can reverse the JSON into WIT. For example:

This command:

go run ./cmd/wit-bindgen-go wit ./testdata/wasi/clocks.wit.json

Generates this:

package wasi:poll

interface poll {
    type pollable = u32
    drop-pollable: func(this: pollable)
    poll-oneoff: func(in: list<pollable>) -> list<bool>
}

world example-world {
    import poll
}


package wasi:clocks

interface monotonic-clock {
    type instant = u64
    use wasi:poll/poll.{pollable}
    now: func() -> instant
    resolution: func() -> instant
    subscribe: func(when: instant, absolute: bool) -> pollable
}

interface timezone {
    use wasi:clocks/wall-clock.{datetime}
    record timezone-display {
        utc-offset: s32,
        name: string,
        in-daylight-saving-time: bool
    }
    display: func(when: datetime) -> timezone-display
    utc-offset: func(when: datetime) -> s32
}

interface wall-clock {
    record datetime {
        seconds: u64,
        nanoseconds: u32
    }
    now: func() -> datetime
    resolution: func() -> datetime
}

world imports {
    import wasi:poll/poll
    import monotonic-clock
    import wall-clock
    import timezone
}

[jordan-rash]

@ydnar if you want more hands on this, im happy to help. Parsing JSON is a lot easier than creating/maintaining a new WIT parser, so I might just archive that effort. That said, happy to help in any way, and most importantly, THANK YOU

[ydnar]

@ydnar if you want more hands on this, im happy to help. Parsing JSON is a lot easier than creating/maintaining a new WIT parser, so I might just archive that effort. That said, happy to help in any way, and most importantly, THANK YOU

The WIT parser you built is undoubtably useful. Maybe at some point we can glue the two together to have another standalone WIT implementation?

As far as wasm-tools-go, pull requests are welcome!

I have a couple things to do before I merge the wit-syntax branch, mainly helpers to determine the nature of a TypeDef (whether it’s a type alias), and synthesizing package-local import paths for use statements.

After that lands, I want to take an initial stab at generating Go types and functions from the WIT tree. My current thinking is to ignore the WIT package → Go package path question for now, and experiment with a couple approaches for mapping the WIT type system to Go (e.g. sealed interfaces vs sealed structs, result<ok, err> as function arguments, etc.

[Mossaka]

Close this issue as it's implemented in wasm-tools bytecodealliance/wasm-tools#1203