WASM plugin

Plugins can be written in WebAssembly (WASM), a portable binary format. This means that plugins can be written in any language that compiles to WASM, like Rust, C, C++, Go, TypeScript, and more. Because WASM based plugins are powered by a programming language, they implicitly support complex business logic and behavior, have access to a sandboxed file system (via WASI), can execute child processes, and much more.

danger

Since our WASM plugin implementation is still experimental, expect breaking changes to occur in non-major releases.

Concepts

Before we begin, let's talk about a few concepts that are critical to WASM and our plugin system.

Virtual paths

WASM by default does not have access to the host file system, but through WASI, we can provide sandboxed access to a pre-defined list of allowed directories. We call this virtual paths, and all paths provided via function input or context use them.

Virtual paths are implemented by mapping a real path (host machine) to a virtual path (guest runtime) using file path prefixes. The following prefixes are currently supported:

Real path	Virtual path
~	/userhome
~/.proto	/proto
CWD	/workspace

For example, from the context of WASM, you may have a virtual path of /proto/tools/node/1.2.3, which simply maps back to ~/.proto/tools/node/1.2.3 on the host machine. However, this should almost always be transparent to you, the developer, and to end users.

However, there may be a few cases where you need access to the real path from WASM, for example, logging or executing commands. For this, the real path can be accessed with the real_path function on the VirtualPath enum (this is a Rust only feature).

input.tool_dir.real_path();

Host environment

Since WASM executes in its own runtime, it does not have access to the current host operating system, architecture, so on and so forth. To bridge this gap, we provide the get_proto_environment function. Learn more about this type.

let env = get_proto_environment()?;

The host operating system and architecture can be accessed with os and arch fields respectively. Both fields are an enum in Rust, or a string in other languages.

if env.os == HostOS::Windows {
    // Windows only
}

if env.arch == HostArch::Arm64 {
    // aarch64 only
}

Furthermore, the user's home directory (~) and proto's root directory (~/.proto) can be accessed with the home_dir and proto_dir fields, both of which are virtual paths.

if env.home_dir.join(some_path).exists() {
    // Do something
}

Host functions

WASM is pretty powerful but it can't do everything since it's sandboxed. To work around this, we provide a mechanism known as host functions, which are functions that are implemented on the host (in Rust), and can be executed from WASM. The following host functions are currently available:

• exec_command - Execute a system command on the host machine, with a provided list of arguments or environment variables.
• get_env_var - Get an environment variable value from the host environment.
• host_log - Log a message to the host's stderr. This acts like tracing logs, and is not a general purpose stdout logger.
• set_env_var - Set an environment variable to the host environment.

To use host functions, you'll need to make them available by registering them at the top of your Rust file (only add the functions you want to use).

#[host_fn]
extern "ExtismHost" {
    fn exec_command(input: Json<ExecCommandInput>) -> Json<ExecCommandOutput>;
    fn get_env_var(key: &str) -> String;
    fn host_log(input: Json<HostLogInput>);
    fn set_env_var(key: &str, value: &str);
}

Environment variables

// Set a value
host_env!("ENV_VAR", "value");

// Get a value (returns an `Option`)
let value = host_env!("ENV_VAR");

Executing commands

// Pipe stdout/stderr
let result = exec_command!("which", ["node"]);
let result = exec_command!(pipe, "npm", ["install"]);

// Inherit stdout/stderr
exec_command!(inherit, "npm", ["install"]);

// Full control
exec_command!(ExecCommandInput {
    command: "npm".into(),
    args: vec!["install".into()],
    env_vars: HashMap::new(),
    stream: false,
});

Logging

host_log!("Some message");
host_log!("Some message with {}", "args");

// With data
host_log!(HostLogInput::Fields {
    message: "Some message with data".into(),
    data: HashMap::from_iter([
        ("data".into(), serde_json::to_value(data)?),
    ]),
});

Tool ID and context

When implementing plugin functions, you'll need to access information about the current tool. To get the current tool ID (the key the plugin was configured with), use the get_tool_id function.

let id = get_tool_id();

Additionally, we also provide what we call the tool context, which is information that is constantly changing depending on the current step or state of proto's execution. The context cannot be accessed with a stand-alone function, and is instead passed as a context field in the input of many plugin functions.

#[plugin_fn]
pub fn download_prebuilt(Json(input): Json<DownloadPrebuiltInput>) -> FnResult<Json<DownloadPrebuiltOutput>> {
    let version = input.context.version;
    // ...
}

The following fields are available on the context object:

• tool_dir - A virtual path to the tool's directory for the current version.
• version - The current version or alias. If not resolved, will be "latest".

caution

The version field is either a fully-qualified semantic version (1.2.3), an alias ("latest", "stable"), or canary ("canary"). Be sure to account for all these variations when implementing plugin functions!

Create a plugin

info

Although plugins can be written in any language that compiles to WASM, we've only tested Rust. The rest of this article assume you're using Rust and Cargo! Refer to Extism's documentation or our official Node.js plugin for other examples.

To start, create a new crate with Cargo:

cargo new plugin --lib
cd plugin

And set the lib type to cdylib, and other settings.

Cargo.toml

[lib]
crate-type = ['cdylib']

[profile.release]
codegen-units = 1
debug = false
lto = true
opt-level = "s"
panic = "abort"

Our Rust plugins are powered by Extism, so lets add their PDK and ours as a dependency.

cargo add extism-pdk proto_pdk

In all Rust files, we can import both PDKs with the following:

src/lib.rs

use extism_pdk::*;
use proto_pdk::*;

We can then build the WASM binary. The file will be available at target/wasm32-wasi/debug/<name>.wasm.

cargo install cargo-wasi
cargo wasi build

Implementing plugin functions

Plugins are powered by a set of functions that are called from the host, and are annotated with #[plugin_fn].

Registering metadata

The first step in a plugin's life-cycle is to register metadata about the plugin with the register_tool function. This function is called immediately after a plugin is loaded at runtime, and must return a human-readable name and plugin type.

#[plugin_fn]
pub fn register_tool(_: ()) -> FnResult<Json<ToolMetadataOutput>> {
    Ok(Json(ToolMetadataOutput {
        name: "Node.js".into(),
        type_of: PluginType::Language,
        plugin_version: Some(env!("CARGO_PKG_VERSION").into()),
        ..ToolMetadataOutput::default()
    }))
}

This function also receives the plugin ID as input, allowing for conditional logic based on the ID. The ID is the key the plugin was configured with, and what is passed to proto commands (e.g. proto install <id>).

#[plugin_fn]
pub fn register_tool(Json(input): Json<ToolMetadataInput>) -> FnResult<Json<ToolMetadataOutput>> {
  input.id
  // ...
}

Downloading pre-builts

Our plugin layer only supports downloading pre-built tools, typically as an archive, and does not support building from source. The download_prebuilt function must be defined, whichs configures how the tool should be downloaded and installed.

The following fields are available:

• archive_prefix - If the tool is distributed as an archive (zip, tar, etc), this is the name of the direct folder within the archive that contains the tool, and will be removed when unpacking the archive. If there is no prefix folder within the archive, this setting can be omitted.
• download_url (required) - A secure URL to download the tool/archive.
• download_name - File name of the archive to download. If not provided, will attempt to extract it from the URL.
• checksum_url - A secure URL to download the checksum file for verification. If the tool does not support checksum verification, this setting can be omitted.
• checksum_public_key - Public key used for verifying checksums. Only used for .minisig files.

#[plugin_fn]
pub fn download_prebuilt(Json(input): Json<DownloadPrebuiltInput>) -> FnResult<Json<DownloadPrebuiltOutput>> {
     let env = get_proto_environment()?;

    check_supported_os_and_arch(
        NAME,
        &env,
        permutations! [
            HostOS::Linux => [HostArch::X64, HostArch::Arm64, HostArch::Arm, HostArch::Powerpc64, HostArch::S390x],
            HostOS::MacOS => [HostArch::X64, HostArch::Arm64],
            HostOS::Windows => [HostArch::X64, HostArch::X86, HostArch::Arm64],
        ],
    )?;

    let version = input.context.version;
    let arch = env.arch;
    let os = env.os;

    let prefix = match os {
        HostOS::Linux => format!("node-v{version}-linux-{arch}"),
        HostOS::MacOS => format!("node-v{version}-darwin-{arch}"),
        HostOS::Windows => format!("node-v{version}-win-{arch}"),
        other => {
            return Err(PluginError::UnsupportedPlatform("Node.js".into(), other.into()))?;
        }
    };

    let filename = if os == HostOS::Windows {
        format!("{prefix}.zip")
    } else {
        format!("{prefix}.tar.xz")
    };

    Ok(Json(DownloadPrebuiltOutput {
        archive_prefix: Some(prefix),
        download_url: format!("https://nodejs.org/dist/v{version}/{filename}"),
        download_name: Some(filename),
        checksum_url: Some(format!("https://nodejs.org/dist/v{version}/SHASUMS256.txt")),
        ..DownloadPrebuiltOutput::default()
    }))
}

Unpacking an archive

Our plugin layer will do its best to detect file extensions, unpack the downloaded file (if an archive), and install the tool to the correct directory. However, we're unable to account for all edge cases, so for situations where the install params above are not sufficient, you may define an unpack_archive function.

This function receives an input with the following fields:

• input_file - Virtual path to the downloaded file. Maps to ~/.proto/temp/<id>/<file>.
• output_dir - Virtual directory to unpack the archive into, or copy the binary to. Maps to ~/.proto/tools/<id>/<version>.

#[plugin_fn]
pub fn unpack_archive(Json(input): Json<UnpackArchiveInput>) -> FnResult<()> {
    untar(input.input_file, input.output_dir)?;
    Ok(())
}

Locating executables

Even though a tool has been installed, we must inform proto of where to find the binary to execute. This can be achieved with the required locate_executables function. The primary field defines the location of the executable, relative from the installation directory.

#[plugin_fn]
pub fn locate_executables(
    Json(_): Json<LocateExecutablesInput>,
) -> FnResult<Json<LocateExecutablesOutput>> {
    let env = get_proto_environment()?;

    Ok(Json(LocateExecutablesOutput {
         primary: Some(ExecutableConfig::new(
            // Helper that chooses between distinct Unix or Windows values
            env.os.for_native("bin/node", "node.exe"),
            // Or the same value with optional Windows extension
            // env.os.get_file_name("node", "exe")
        )),
        ..LocateExecutablesOutput::default()
    }))
}

Furthermore, the locate_executables function can define a list of lookups for the globals installation directory. proto will loop through each lookup, and return the first directory that exists on the current file system. proto will also expand environment variables in the format of $VAR_NAME. If a variable is not defined, or has an empty value, the lookup will be skipped. To demonstrate this, we'll use Deno.

#[plugin_fn]
pub fn locate_executables(
    Json(_): Json<LocateExecutablesInput>,
) -> FnResult<Json<LocateExecutablesOutput>> {
    let env = get_proto_environment()?;

    Ok(Json(LocateExecutablesOutput {
        globals_lookup_dirs: vec!["$DENO_INSTALL_ROOT/bin".into(), "$HOME/.deno/bin".into()],
        // ...
        ..LocateExecutablesOutput::default()
    }))
}

Loading and resolving versions

Now that the tool can be downloaded and installed, we must configure how to resolve available versions to actually be installed. To provide a list of versions and language specific aliases, the load_versions function must be defined.

#[plugin_fn]
pub fn load_versions(Json(_): Json<LoadVersionsInput>) -> FnResult<Json<LoadVersionsOutput>> {
    let mut output = LoadVersionsOutput::default();
    let response: Vec<NodeDistVersion> = fetch_url("https://nodejs.org/dist/index.json")?;

    for (index, item) in response.iter().enumerate() {
        let version = Version::parse(&item.version[1..])?; // Starts with v

        if index == 0 {
            output.latest = Some(version.clone());
        }

        output.versions.push(version);
    }

    Ok(Json(output))
}

Furthermore, we support an optional function named resolve_version, that can be defined to intercept the version resolution process. This function receives an input with an initial candidate, either an alias or version, and can replace it with a new candidate. The candidate must be a valid alias or version as defined in load_versions.

#[plugin_fn]
pub fn resolve_version(
    Json(input): Json<ResolveVersionInput>,
) -> FnResult<Json<ResolveVersionOutput>> {
    let mut output = ResolveVersionOutput::default();

    if let UnresolvedVersionSpec::Alias(alias) = input.initial {
        let candidate = if alias == "node" {
            "latest"
        } else if alias == "lts-*" || alias == "lts/*" {
            "stable"
        } else if alias.starts_with("lts-") || alias.starts_with("lts/") {
            &alias[4..]
        } else {
            return Ok(Json(output));
        };

        output.candidate = Some(UnresolvedVersionSpec::Alias(candidate.to_owned()));
    }

    Ok(Json(output))
}

Detecting versions

And lastly, we can configure how to detect a version contextually at runtime, using the detect_version_files function and optional parse_version_file function. The detect_version_files function can return a list of files to locate within a directory.

#[plugin_fn]
pub fn detect_version_files(_: ()) -> FnResult<Json<DetectVersionOutput>> {
    Ok(Json(DetectVersionOutput {
        files: vec![
            ".nvmrc".into(),
            ".node-version".into(),
            "package.json".into(),
        ],
    }))
}

By default our plugin layer will assume the version file's contents contain the literal version, and nothing else, like "1.2.3". If any of the files in the detect_version_files list require custom parsing (for example, package.json above), you can define the parse_version_file function.

This function receives the file name and contents as input, and must return the parsed version (if applicable).

#[plugin_fn]
pub fn parse_version_file(Json(input): Json<ParseVersionFileInput>) -> FnResult<Json<ParseVersionFileOutput>> {
    let mut version = None;

    if input.file == "package.json" {
        let json: PackageJson = serde_json::from_str(&input.content)?;

        if let Some(engines) = json.engines {
            if let Some(constraint) = engines.get("node") {
                version = Some(UnresolvedVersionSpec::parse(constraint)?);
            }
        }
    } else {
        version = Some(UnresolvedVersionSpec::parse(input.content.trim())?);
    }

    Ok(Json(ParseVersionFileOutput { version }))
}

Installing and uninstalling globals

Most languages support the concept of installing packages/dependencies globally, and making them available on PATH. proto supports this with the proto install-global and proto uninstall-global commands, which are simple convenience wrappers around the native binary.

From the context of the plugin, how globals are installed and uninstalled is implemented with the install_global and uninstall_global functions respectively. Both functions receive the dependency to install as input.

#[plugin_fn]
pub fn install_global(
    Json(input): Json<InstallGlobalInput>,
) -> FnResult<Json<InstallGlobalOutput>> {
    let result = exec_command!(inherit, "npm", ["install", "--global", &input.dependency]);

    Ok(Json(InstallGlobalOutput::from_exec_command(result)))
}

#[plugin_fn]
pub fn uninstall_global(
    Json(input): Json<UninstallGlobalInput>,
) -> FnResult<Json<UninstallGlobalOutput>> {
    let result = exec_command!(inherit, "npm", ["uninstall", "--global", &input.dependency]);

    Ok(Json(UninstallGlobalOutput::from_exec_command(result)))
}

Testing

The best way to test the plugin is to execute it through proto directly. To do this, you'll need to configure a .prototools file at the root of your plugin's repository that maps the plugin to a debug build:

[plugins]
<id> = "source:target/wasm32-wasi/debug/<name>.wasm"

And everytime you make a change to the plugin, you'll need to rebuild it with:

cargo wasi build

With these 2 pieces in place, you can now execute proto commands. Be sure you're running them from the directory with the .prototools file, and that you're passing --log trace. Logs are extremely helpful for figuring out what's going on.

proto --log trace install <id>
proto --log trace list-remote <id>
...

Unit tests

Testing WASM plugins is a bit tricky, but we've taken it upon ourselves to streamline this process as much as possible with built-in test utilities, and Rust macros for generating common test cases. To begin, install all necessary development dependencies:

cargo add --dev proto_pdk_test_utils starbase_sandbox tokio

And as mentioned above, everytime you make a change to the plugin, you'll need to rebuild it with:

cargo wasi build

Testing plugin functions

The common test case is simply calling plugin functions with a provided input and asserting the output is correct. This can be achieved by creating a plugin instance with create_plugin and calling the appropriate method.

use proto_pdk_test_utils::*;
use starbase_sandbox::create_empty_sandbox;

#[test]
fn registers_metadata() {
    let sandbox = create_empty_sandbox();
    let plugin = create_plugin("id", sandbox.path());

    assert_eq!(
        plugin.register_tool(ToolMetadataInput::default()),
        ToolMetadataOutput {
            name: "Name".into(),
            ..ToolMetadataOutput::default()
        }
    );
}

info

We suggest using this pattern for static functions that return a deterministic output from a provided input, and not for dynamic functions that make HTTP requests or execute host commands.

Generating cases from macros

To reduce the burden of writing custom tests for common flows, like downloading a pre-built, resolving versions, and generating shims, we provide a set of Rust decl macros that will generate the tests for you.

To test downloading and installing, use generate_download_install_tests!. This macro requires a plugin ID and a real version to test with.

use proto_pdk_test_utils::*;

generate_download_install_tests!("id", "1.2.3");

To test version resolving, use generate_resolve_versions_tests!. This macro requires a plugin ID, and a mapping of version/aliases assertions to expectations.

generate_resolve_versions_tests!("id", {
    "0.4" => "0.4.12",
    "0.5.1" => "0.5.1",
    "stable" => "1.0.0",
});

To test installing and uninstalling globals, use generate_globals_test!. This macro requires a plugin ID, the dependency to install, and an optional environment variable to the globals directory.

// Doesn't support all use cases! If this doesn't work, implement a test case manually.
generate_globals_test!("id", "dependency", "GLOBAL_INSTALL_ROOT");

And lastly, to test shims, use generate_shims_test!. This requires a plugin ID, but also supports additional arguments when creating more than 1 shim. This macro generates snapshots using Insta.

// Only the single binary
generate_shims_test!("id");

// When creating alternate/additional globals
generate_shims_test!("id", ["other", "another"]);

Building and publishing

At this point, you should have a fully working WASM plugin, but to make it available to downstream proto users, you'll still need to build and make the .wasm file available. The easiest solution is to publish a GitHub release and include the .wasm file as an asset.

Building, optimizing, and stripping

WASM files are pretty fat, even when compiling in release mode. To reduce the size of these files, we can use wasm-opt and wasm-strip, both of which are provided by the WebAssembly group. The following script is what we use to build our own plugins.

info

This functionality is natively supported in our build-proto-plugin GitHub Action!

build-wasm

#!/usr/bin/env bash

target="${CARGO_TARGET_DIR:-target}"
input="$target/wasm32-wasi/release/$1.wasm"
output="$target/wasm32-wasi/$1.wasm"

echo "Building"

cargo build --target wasm32-wasi --release

echo "Optimizing"

# https://github.com/WebAssembly/binaryen
~/binaryen/bin/wasm-opt -Os "$input" --output "$output"

echo "Stripping"

# https://github.com/WebAssembly/wabt
~/wabt/bin/wasm-strip "$output"

Manually create releases

When your plugin is ready to be published, you can create a release on GitHub using the following steps.

1. Tag the release and push to GitHub.

git tag v0.0.1
git push --tags

2. Build a release version of the plugin using the build-wasm script above. The file will be available at target/wasm32-wasi/<name>.wasm.

build-wasm <name>

3. In GitHub, navigate to the tags page, find the new tag, create a new release, and attach the built file as an asset.

Automate releases

If you're using GitHub Actions, you can automate the release process with our official build-proto-plugin action.

1. Create a new workflow file at .github/workflows/release.yml. Refer to the link above for a working example.

2. Tag the release and push to GitHub.

git tag v0.0.1
git push --tags

3. The action will automatically build the plugin, create a release, and attach the built file as an asset.

Resources

Some helpful resources for learning about and building plugins.

• Official proto WASM plugins
  • Bun
  • Deno
  • Go
  • Node
  • npm, pnpm, yarn
  • Python
  • Rust
  • Schema
• Plugin development kit
  • proto_pdk docs
  • proto_pdk_test_utils docs