Skip to main content

WASM plugin

If you want more control over how your tool works, a WASM plugin is the way to go.

success

Refer to our official WASM guide for more information on how our WASM plugins work, critical concepts to know, and more. Once you have a good understanding, you may continue this proto specific guide.

Concepts​

The following concepts are unique to proto, but be sure to also read about the general concepts in our WASM plugins guide.

Tool context​

For plugin functions, we provide what we call the tool context, which is information that is constantly changing depending on the current step or state of proto's execution. The context cannot be accessed with a stand-alone function, and is instead passed as a context field in the input of many plugin functions.

#[plugin_fn]
pub fn download_prebuilt(Json(input): Json<DownloadPrebuiltInput>) -> FnResult<Json<DownloadPrebuiltOutput>> {
    let version = input.context.version;
    // ...
}

The following fields are available on the context object:

  • proto_version - The version of proto executing the plugin. Note that this version may be for the proto_core crate, and not the CLI. Patch numbers will drift, but major and minor numbers should be in sync.
  • tool_dir - A virtual path to the tool's directory for the current version.
  • version - The current version or alias. If not resolved, will be "latest".
caution

The version field is either a fully-qualified version (1.2.3), an alias ("latest", "stable"), or canary ("canary"). Be sure to account for all these variations when implementing plugin functions!

Tool configuration​

Users can configure tools through the [tools.*] section of their .prototools, which can then be accessed within the WASM plugin using the get_tool_config function.

let config = get_tool_config::<NodeConfig>()?;
config.bundled_npm;

This function requires a struct to deserialize into. It should implement Default, enable serde defaults, and map keys from kebab-case. If you want to error on unknown settings, also enable deny_unknown_fields.

#[derive(Debug, Default, serde::Deserialize)]
#[serde(default, deny_unknown_fields, rename_all = "kebab-case")]
struct NodeConfig {
    pub bundled_npm: bool,
    pub intercept_globals: bool,
}

Creating a plugin​

success

Refer to our official WASM guide for steps on how to create a Rust based plugin.

Implementing plugin functions​

Plugins are powered by a set of functions that are called from the host, and are annotated with #[plugin_fn].

Registering metadata​

The first step in a plugin's life-cycle is to register metadata about the plugin with the register_tool function. This function is called immediately after a plugin is loaded at runtime, and must return a human-readable name and plugin type.

#[plugin_fn]
pub fn register_tool(_: ()) -> FnResult<Json<ToolMetadataOutput>> {
    Ok(Json(ToolMetadataOutput {
        name: "Node.js".into(),
        type_of: PluginType::Language,
        minimum_proto_version: Some(Version::new(0, 42, 0)),
        plugin_version: Version::parse(env!("CARGO_PKG_VERSION")).ok(),
        ..ToolMetadataOutput::default()
    }))
}

This function also receives the plugin ID as input, allowing for conditional logic based on the ID. The ID is the key the plugin was configured with, and what is passed to proto commands (e.g. proto install <id>).

#[plugin_fn]
pub fn register_tool(Json(input): Json<ToolMetadataInput>) -> FnResult<Json<ToolMetadataOutput>> {
  input.id
  // ...
}

Configuration schemav0.40.0​

If you are using tool configuration, you can register the shape of the configuration using the schematic crate. This shape will be used to generate outputs such as JSON schemas, or TypeScript types.

#[plugin_fn]
pub fn register_tool(_: ()) -> FnResult<Json<ToolMetadataOutput>> {
    Ok(Json(ToolMetadataOutput {
        // ...
        config_schema: Some(schematic::SchemaBuilder::generate::<NodeConfig>()),
    }))
}

Schematic is a heavy library, so we suggest adding the dependency like so:

[dependencies]
schematic = { version = "*", default-features = false, features = ["schema"] }

Downloading pre-builts​

Our plugin layer only supports downloading pre-built tools, typically as an archive, and does not support building from source. The download_prebuilt function must be defined, whichs configures how the tool should be downloaded and installed.

The following fields are available:

  • archive_prefix - If the tool is distributed as an archive (zip, tar, etc), this is the name of the direct folder within the archive that contains the tool, and will be removed when unpacking the archive. If there is no prefix folder within the archive, this setting can be omitted.
  • download_url (required) - A secure URL to download the tool/archive.
  • download_name - File name of the archive to download. If not provided, will attempt to extract it from the URL.
  • checksum_url - A secure URL to download the checksum file for verification. If the tool does not support checksum verification, this setting can be omitted.
  • checksum_public_key - Public key used for verifying checksums. Only used for .minisig files.
#[plugin_fn]
pub fn download_prebuilt(Json(input): Json<DownloadPrebuiltInput>) -> FnResult<Json<DownloadPrebuiltOutput>> {
     let env = get_host_environment()?;

    check_supported_os_and_arch(
        NAME,
        &env,
        permutations! [
            HostOS::Linux => [HostArch::X64, HostArch::Arm64, HostArch::Arm, HostArch::Powerpc64, HostArch::S390x],
            HostOS::MacOS => [HostArch::X64, HostArch::Arm64],
            HostOS::Windows => [HostArch::X64, HostArch::X86, HostArch::Arm64],
        ],
    )?;

    let version = input.context.version;
    let arch = env.arch;
    let os = env.os;

    let prefix = match os {
        HostOS::Linux => format!("node-v{version}-linux-{arch}"),
        HostOS::MacOS => format!("node-v{version}-darwin-{arch}"),
        HostOS::Windows => format!("node-v{version}-win-{arch}"),
        other => {
            return Err(PluginError::UnsupportedPlatform("Node.js".into(), other.into()))?;
        }
    };

    let filename = if os == HostOS::Windows {
        format!("{prefix}.zip")
    } else {
        format!("{prefix}.tar.xz")
    };

    Ok(Json(DownloadPrebuiltOutput {
        archive_prefix: Some(prefix),
        download_url: format!("https://nodejs.org/dist/v{version}/{filename}"),
        download_name: Some(filename),
        checksum_url: Some(format!("https://nodejs.org/dist/v{version}/SHASUMS256.txt")),
        ..DownloadPrebuiltOutput::default()
    }))
}

Unpacking an archive​

Our plugin layer will do its best to detect file extensions, unpack the downloaded file (if an archive), and install the tool to the correct directory. However, we're unable to account for all edge cases, so for situations where the install params above are not sufficient, you may define an unpack_archive function.

This function receives an input with the following fields:

  • input_file - Virtual path to the downloaded file. Maps to ~/.proto/temp/<id>/<file>.
  • output_dir - Virtual directory to unpack the archive into, or copy the binary to. Maps to ~/.proto/tools/<id>/<version>.
#[plugin_fn]
pub fn unpack_archive(Json(input): Json<UnpackArchiveInput>) -> FnResult<()> {
    untar(input.input_file, input.output_dir)?;
    Ok(())
}

Locating executables​

Even though a tool has been installed, we must inform proto of where to find executables. This can be achieved with the required locate_executables function. The exes field defines the location of the executables, relative from the installation directory.

#[plugin_fn]
pub fn locate_executables(
    Json(_): Json<LocateExecutablesInput>,
) -> FnResult<Json<LocateExecutablesOutput>> {
    let env = get_host_environment()?;

    Ok(Json(LocateExecutablesOutput {
        exes: HashMap::from_iter([
            // Primary
            (
                "node".into(),
                ExecutableConfig::new_primary(
                    // Helper that chooses between distinct Unix or Windows values
                    env.os.for_native("bin/node", "node.exe"),
                    // Or the same value with optional Windows extension
                    // env.os.get_file_name("node", "exe")
                )
            ),
            // Secondary
            (
                "corepack".into(),
                ExecutableConfig::new(env.os.for_native("bin/corepack", "corepack.exe"))
            ),
        ]),
        ..LocateExecutablesOutput::default()
    }))
}

The main executable of the tool must be marked as primary, either with ExecutableConfig::new_primary, or setting the ExecutableConfig.primary field to true.

Furthermore, the locate_executables function can define a list of lookups for the globals installation directory. proto will loop through each lookup, and return the first directory that exists on the current file system. proto will also expand environment variables in the format of $VAR_NAME. If a variable is not defined, or has an empty value, the lookup will be skipped. To demonstrate this, we'll use Deno.

#[plugin_fn]
pub fn locate_executables(
    Json(_): Json<LocateExecutablesInput>,
) -> FnResult<Json<LocateExecutablesOutput>> {
    let env = get_host_environment()?;

    Ok(Json(LocateExecutablesOutput {
        globals_lookup_dirs: vec!["$DENO_INSTALL_ROOT/bin".into(), "$HOME/.deno/bin".into()],
        // ...
        ..LocateExecutablesOutput::default()
    }))
}

Loading and resolving versions​

Now that the tool can be downloaded and installed, we must configure how to resolve available versions to actually be installed. To provide a list of versions and language specific aliases, the load_versions function must be defined.

#[plugin_fn]
pub fn load_versions(Json(_): Json<LoadVersionsInput>) -> FnResult<Json<LoadVersionsOutput>> {
    let mut output = LoadVersionsOutput::default();
    let response: Vec<NodeDistVersion> = fetch_json("https://nodejs.org/dist/index.json")?;

    for (index, item) in response.iter().enumerate() {
        let version = VersionSpec::parse(&item.version[1..])?; // Starts with v

        if index == 0 {
            output.latest = Some(version.clone());
        }

        output.versions.push(version);
    }

    Ok(Json(output))
}

Furthermore, we support an optional function named resolve_version, that can be defined to intercept the version resolution process. This function receives an input with an initial candidate, either an alias or version, and can replace it with a new candidate. The candidate must be a valid alias or version as defined in load_versions.

#[plugin_fn]
pub fn resolve_version(
    Json(input): Json<ResolveVersionInput>,
) -> FnResult<Json<ResolveVersionOutput>> {
    let mut output = ResolveVersionOutput::default();

    if let UnresolvedVersionSpec::Alias(alias) = input.initial {
        let candidate = if alias == "node" {
            "latest"
        } else if alias == "lts-*" || alias == "lts/*" {
            "stable"
        } else if alias.starts_with("lts-") || alias.starts_with("lts/") {
            &alias[4..]
        } else {
            return Ok(Json(output));
        };

        output.candidate = Some(UnresolvedVersionSpec::Alias(candidate.to_owned()));
    }

    Ok(Json(output))
}

Detecting versions​

And lastly, we can configure how to detect a version contextually at runtime, using the detect_version_files function and optional parse_version_file function. The detect_version_files function can return a list of files to locate within a directory.

#[plugin_fn]
pub fn detect_version_files(_: ()) -> FnResult<Json<DetectVersionOutput>> {
    Ok(Json(DetectVersionOutput {
        files: vec![
            ".nvmrc".into(),
            ".node-version".into(),
            "package.json".into(),
        ],
        ignore: vec!["node_modules".into()],
    }))
}

By default our plugin layer will assume the version file's contents contain the literal version, and nothing else, like "1.2.3". If any of the files in the detect_version_files list require custom parsing (for example, package.json above), you can define the parse_version_file function.

This function receives the file name and contents as input, and must return the parsed version (if applicable).

#[plugin_fn]
pub fn parse_version_file(Json(input): Json<ParseVersionFileInput>) -> FnResult<Json<ParseVersionFileOutput>> {
    let mut version = None;

    if input.file == "package.json" {
        let json: PackageJson = serde_json::from_str(&input.content)?;

        if let Some(engines) = json.engines {
            if let Some(constraint) = engines.get("node") {
                version = Some(UnresolvedVersionSpec::parse(constraint)?);
            }
        }
    } else {
        version = Some(UnresolvedVersionSpec::parse(input.content.trim())?);
    }

    Ok(Json(ParseVersionFileOutput { version }))
}

Testing​

The best way to test the plugin is to execute it through proto directly. To do this, you'll need to configure a .prototools file at the root of your plugin's repository that maps the plugin to a debug build:

[plugins]
<id> = "file://./target/wasm32-wasi/debug/<name>.wasm"

And everytime you make a change to the plugin, you'll need to rebuild it with:

cargo wasi build

With these 2 pieces in place, you can now execute proto commands. Be sure you're running them from the directory with the .prototools file, and that you're passing --log trace. Logs are extremely helpful for figuring out what's going on.

proto --log trace install <id>
proto --log trace list-remote <id>
...

Unit tests​

Testing WASM plugins is a bit tricky, but we've taken it upon ourselves to streamline this process as much as possible with built-in test utilities, and Rust macros for generating common test cases. To begin, install all necessary development dependencies:

cargo add --dev proto_pdk_test_utils starbase_sandbox tokio

And as mentioned above, everytime you make a change to the plugin, you'll need to rebuild it with:

cargo wasi build

Testing plugin functions​

The common test case is simply calling plugin functions with a provided input and asserting the output is correct. This can be achieved by creating a plugin instance with create_plugin and calling the appropriate method.

use proto_pdk_test_utils::*;
use starbase_sandbox::create_empty_sandbox;

#[tokio::test(flavor = "multi_thread")]
async fn registers_metadata() {
    let sandbox = create_empty_proto_sandbox();
    let plugin = sandbox.create_plugin("id").await;

    assert_eq!(
        plugin.register_tool(ToolMetadataInput::default()).await,
        ToolMetadataOutput {
            name: "Name".into(),
            ..ToolMetadataOutput::default()
        }
    );
}
info

We suggest using this pattern for static functions that return a deterministic output from a provided input, and not for dynamic functions that make HTTP requests or execute host commands.

Generating cases from macros​

To reduce the burden of writing custom tests for common flows, like downloading a pre-built, resolving versions, and generating shims, we provide a set of Rust decl macros that will generate the tests for you.

To test downloading and installing, use generate_download_install_tests!. This macro requires a plugin ID and a real version to test with.

use proto_pdk_test_utils::*;

generate_download_install_tests!("id", "1.2.3");

To test version resolving, use generate_resolve_versions_tests!. This macro requires a plugin ID, and a mapping of version/aliases assertions to expectations.

generate_resolve_versions_tests!("id", {
    "0.4" => "0.4.12",
    "0.5.1" => "0.5.1",
    "stable" => "1.0.0",
});

To test installing and uninstalling globals, use generate_globals_test!. This macro requires a plugin ID, the dependency to install, and an optional environment variable to the globals directory.

// Doesn't support all use cases! If this doesn't work, implement a test case manually.
generate_globals_test!("id", "dependency", "GLOBAL_INSTALL_ROOT");

And lastly, to test shims, use generate_shims_test!. This requires a plugin ID and a list of shim file names. This macro generates snapshots using Insta.

generate_shims_test!("id", ["primary", "secondary"]);

Building and publishing​

success

Refer to our official WASM guide for steps on how to build and publish your plugin.

Resources​

Some helpful resources for learning about and building plugins.