WASM plugins

moon and proto plugins can be written in WebAssembly (WASM), a portable binary format. This means that plugins can be written in any language that compiles to WASM, like Rust, C, C++, Go, TypeScript, and more. Because WASM based plugins are powered by a programming language, they implicitly support complex business logic and behavior, have access to a sandboxed file system (via WASI), can execute child processes, and much more.

danger

Since our WASM plugin implementations are still experimental, expect breaking changes to occur in non-major releases.

Powered by Extism

Our WASM plugin system is powered by Extism, a Rust-based cross-language framework for building WASM plugins under a unified guest and host API. Under the hood, Extism uses wasmtime as its WASM runtime.

For the most part, you do not need to know about Extism's host SDK, as we have implemented the bulk of it within moon and proto directly. However, you should be familiar with the guest PDKs, as this is what you'll be using to implement Rust-based plugins. We suggest reading the following material:

• Plugin development kits (PDKs)
• The extism-pdk Rust crate
• Host functions (how they work)

Concepts

Before we begin, let's talk about a few concepts that are critical to WASM and our plugin systems.

Plugin identifier

When implementing plugin functions, you'll need to access information about the current plugin. To get the current plugin identifier (the key the plugin was configured with), use the get_plugin_id function.

let id = get_plugin_id();

Virtual paths

WASM by default does not have access to the host file system, but through WASI, we can provide sandboxed access to a pre-defined list of allowed directories. We call these virtual paths, and all paths provided via function input or context use them.

Virtual paths are implemented by mapping a real path (host machine) to a virtual path (guest runtime) using file path prefixes. The following prefixes are currently supported:

Real path	Virtual path	Only for
~	/userhome	~
~/.proto	/proto	~
~/.moon	/moon	moon
Working directory	/cwd	~
moon workspace	/workspace	moon

For example, from the context of WASM, you may have a virtual path of /proto/tools/node/1.2.3, which simply maps back to ~/.proto/tools/node/1.2.3 on the host machine. However, this should almost always be transparent to you, the developer, and to end users.

However, there may be a few cases where you need access to the real path from WASM, for example, logging or executing commands. For this, the real path can be accessed with the real_path function on the VirtualPath enum (this is a Rust only feature).

virtual_path.real_path();

File system caveats

When working with the file system from the context of WASM, there are a few caveats to be aware of.

• All fs calls must use the virtual path. Real paths will error.
• Paths not white listed (using prefixes above) will error.
• Changing file permissions is not supported (on Unix and Windows).
  • This is because WASI does not support this.
  • This also means operations like unpacking archives is not possible.

Host environment

Since WASM executes in its own runtime, it does not have access to the current host operating system, architecture, so on and so forth. To bridge this gap, we provide the get_host_environment function. Learn more about this type.

let env = get_host_environment()?;

The host operating system and architecture can be accessed with os and arch fields respectively. Both fields are an enum in Rust, or a string in other languages.

if env.os == HostOS::Windows {
    // Windows only
}

if env.arch == HostArch::Arm64 {
    // aarch64 only
}

Furthermore, the user's home directory (~) can be accessed with the home_dir field, which is a virtual path.

if env.home_dir.join(some_path).exists() {
    // Do something
}

Host functions & macros

WASM is pretty powerful but it can't do everything since it's sandboxed. To work around this, we provide a mechanism known as host functions, which are functions that are implemented on the host (in Rust), and can be executed from WASM. The following host functions are currently available:

• exec_command - Execute a system command on the host machine, with a provided list of arguments or environment variables.
• from_virtual_path - Converts a virtual path into a real path.
• get_env_var - Get an environment variable value from the host environment.
• host_log - Log an stdout, stderr, or tracing message to the host's terminal.
• send_request - Requests a URL on the host machine using a Rust-based HTTP client (not WASM).
• set_env_var - Set an environment variable to the host environment.
• to_virtual_path - Converts a real path into a virtual path.

To use host functions, you'll need to make them available by registering them at the top of your Rust file (only add the functions you want to use) using the extism-pdk crate.

use extism_pdk::*;

#[host_fn]
extern "ExtismHost" {
    fn exec_command(input: Json<ExecCommandInput>) -> Json<ExecCommandOutput>;
    fn from_virtual_path(path: String) -> String;
    fn get_env_var(key: String) -> String;
    fn host_log(input: Json<HostLogInput>);
    fn send_request(input: Json<SendRequestInput>) -> Json<SendRequestOutput>;
    fn set_env_var(key: String, value: String);
    fn to_virtual_path(path: String) -> String;
}

info

To simplify development, we provide built-in macros for the host functions above. Continue reading for more information on these macros.

Converting paths

When working with virtual paths, you may need to convert them to real paths, and vice versa. The virtual_path! and real_path! macros can be used for such situations, which use the to_virtual_path and from_virtual_path host functions respectively.

// For strings
let virt = virtual_path!("/some/real/path");
let real = real_path!("/some/virtual/path");

// For `Path` and `PathBuf`
let virt = virtual_path!(buf, PathBuf::from("/some/real/path"));
let real = real_path!(buf, PathBuf::from("/some/virtual/path"));

Environment variables

The host_env! macro can be used to read and write environment variables on the host, using the set_env_var and get_env_var host functions respectively.

// Set a value
host_env!("ENV_VAR", "value");

// Append to path
host_env!("PATH", "/userhome/some/virtual/path");

// Get a value (returns an `Option`)
let value = host_env!("ENV_VAR");

Executing commands

The exec_command! macro can be used to execute a command on the host, using the exec_command host function. If the command does not exist on PATH, an error is thrown.

This macros supports three modes: pipe, inherit, and raw (returns Result). If you want full control, use the input mode and provide ExecCommandInput.

// Pipe stdout/stderr
let output = exec_command!("which", ["node"]);
let output = exec_command!(pipe, "npm", ["install"]);

// Inherit stdout/stderr
exec_command!(inherit, "npm", ["install"]);

// Full control
exec_command!(input, ExecCommandInput {
    command: "npm".into(),
    args: vec!["install".into()],
    ..ExecCommandInput::default()
});

Sending requests

The send_request macro can be used to request a URL on the host, instead of from WASM, allowing it to use the same HTTP client as the host CLI. This macro returns a response object, with the raw body in bytes, and the status code.

let response = send_request!("https://some.com/url/to/fetch");

if response.status == 200 {
  let json = response.json::<T>()?;
  let text = response.text()?;
} else {
  // Error!
}

To simplify the handling of requests -> responses, we also provide the fetch_bytes, fetch_json, and fetch_text functions.

let json: T = fetch_json("https://some.com/url/to/fetch.json")?;

Only GET requests are supported.

Logging

The host_log! macro can be used to write stdout or stderr messages to the host's terminal, using the host_log host function. It supports the same argument patterns as format!.

If you want full control, like providing data/fields, use the input mode and provide HostLogInput.

host_log!(stdout, "Some message");
host_log!(stderr, "Some message with {}", "args");

// With data
host_log!(input, HostLogInput {
    message: "Some message with data".into(),
    data: HashMap::from_iter([
        ("data".into(), serde_json::to_value(data)?),
    ]),
    target: HostLogTarget::Stderr,
});

Furthermore, the extism-pdk crate provides a handful of macros for writing level-based messages that'll appear in the host's terminal when --log is enabled in the CLI. These also support arguments.

debug!("This is a debug message");
info!("Something informational happened");
warn!("Proceed with caution");
error!("Oh no, something went wrong");

Configuring plugin locations

To use a WASM plugin, it'll need to be configured in both moon and proto. Luckily both tools use a similar approach for configuring plugins called the plugin locator. A locator string is composed of 2 parts separated by ://, the former is the protocol, and the latter is the location.

"<protocol>://<location>"

The following locator patterns are supported:

file

The file:// protocol represents a file path, either absolute or relative (from the current configuration file).

# Relative
"file://./path/to/example.wasm"

# Absolute
"file:///root/path/to/example.wasm"

github

The github:// protocol can be used to target and download an asset from a specific GitHub release. The location must be an organization + repository slug (owner/repo), and the release must have a .wasm asset available to download.

"github://moonrepo/example-repo"

If you are targeting releases in a monorepo, you can append the project name after the repository. The project name will be used as a prefix for tags, and will match <name>@v?<version> or <name>-v?<version> based tags.

"github://moonrepo/example-repo/project-name"

By default, the latest release will be used and cached for 7 days. If you'd prefer to target a specific release (preferred), append the release tag to the end of the location.

"github://moonrepo/example-repo@v1.2.3"

This strategy is powered by the GitHub API and is subject to rate limiting. If running in a CI environment, we suggesting setting a GITHUB_TOKEN environment variable to authorize API requests with. If using GitHub Actions, it's as simple as:

# In some job or step...
env:
  GITHUB_TOKEN: '${{ secrets.GITHUB_TOKEN }}'

https

The https:// protocol is your standard URL, and must point to an absolute file path. Files will be downloaded to ~/.moon/plugins or ~/.proto/plugins. Non-secure URLs are not supported!

"https://domain.com/path/to/plugins/example.wasm"

Creating a plugin

info

Although plugins can be written in any language that compiles to WASM, we've only tested Rust. The rest of this article assume you're using Rust and Cargo! Refer to Extism's documentation for other examples.

To start, create a new crate with Cargo:

cargo new plugin --lib
cd plugin

Set the lib type to cdylib, and provide other required settings.

Cargo.toml

[package]
name = "example_plugin"
version = "0.0.1"
edition = "2021"
publish = false

[lib]
crate-type = ['cdylib']

[profile.release]
codegen-units = 1
debug = false
lto = true
opt-level = "s"
panic = "abort"

Our Rust plugins are powered by Extism, so lets add their PDK and ours as a dependency.

cargo add extism-pdk

# For proto
cargo add proto_pdk

# For moon
cargo add moon_pdk

In all Rust files, we can import all the PDKs with the following:

src/lib.rs

use extism_pdk::*;

We can then build the WASM binary. The file will be available at target/wasm32-wasi/debug/<name>.wasm.

cargo install cargo-wasi
cargo wasi build

Building and publishing

At this point, you should have a fully working WASM plugin, but to make it available to the community, you'll still need to build and make the .wasm file available. The easiest solution is to publish a GitHub release and include the .wasm file as an asset.

Building, optimizing, and stripping

WASM files are pretty fat, even when compiling in release mode. To reduce the size of these files, we can use wasm-opt and wasm-strip, both of which are provided by the WebAssembly group. The following script is what we use to build our own plugins.

info

This functionality is natively supported in our moonrepo/build-wasm-plugin GitHub Action!

build-wasm

#!/usr/bin/env bash

target="${CARGO_TARGET_DIR:-target}"
input="$target/wasm32-wasi/release/$1.wasm"
output="$target/wasm32-wasi/$1.wasm"

echo "Building"

cargo build --target wasm32-wasi --release

echo "Optimizing"

# https://github.com/WebAssembly/binaryen
~/binaryen/bin/wasm-opt -Os "$input" --output "$output"

echo "Stripping"

# https://github.com/WebAssembly/wabt
~/wabt/bin/wasm-strip "$output"

Manually create releases

When your plugin is ready to be published, you can create a release on GitHub using the following steps.

1. Tag the release and push to GitHub.

git tag v0.0.1
git push --tags

2. Build a release version of the plugin using the build-wasm script above. The file will be available at target/wasm32-wasi/<name>.wasm.

build-wasm <name>

3. In GitHub, navigate to the tags page, find the new tag, create a new release, and attach the built file as an asset.

Automate releases

If you're using GitHub Actions, you can automate the release process with our official moonrepo/build-wasm-plugin action.

1. Create a new workflow file at .github/workflows/release.yml. Refer to the link above for a working example.

2. Tag the release and push to GitHub.

# In a polyrepo
git tag v0.0.1

# In a monorepo
git tag example_plugin-v0.0.1

# Push the tags
git push --tags

3. The action will automatically build the plugin, create a release, and attach the built file as an asset.