Skip to main content

moon v1.29 - Improved affected tracking, experimental Pkl configuration, and more

· 9 min read
Miles Johnson
Miles Johnson
Founder, developer

In this release, we're excited to introduce an improved affected tracker and a new (but experimental) configuration format!

New affected projects tracker​

We've received a lot of feedback that our affected projects and tasks logic works differently across commands, or that it's hard to understand why something is affected or not affected. We wanted to add more clarity around affected projects, so have implemented a new affected tracker.

This new tracker includes a ton of new logging that we believe will answer the "why". For example, once the tracker has finished tracking, we'll log all affected projects and tasks, and what marked their affected state.

[DEBUG] moon_affected::affected_tracker  Project website is affected by  files=["website/blog/2024-10-01_moon-v1.29.mdx"] upstream=[] downstream=[] other=false
[DEBUG] moon_affected::affected_tracker  Project runtime is affected by files=[] upstream=[] downstream=["website"] other=false
[DEBUG] moon_affected::affected_tracker  Project types is affected by  files=[] upstream=[] downstream=["website", "runtime"] other=false
[DEBUG] moon_affected::affected_tracker  Task runtime:build is affected by  env=[] files=[] upstream=[] downstream=["website:start", "website:build"] other=false
[DEBUG] moon_affected::affected_tracker  Task website:start is affected by  env=[] files=["website/blog/2024-10-01_moon-v1.29.mdx"] upstream=[] downstream=[] other=false
[DEBUG] moon_affected::affected_tracker  Task types:build is affected by  env=[] files=[] upstream=[] downstream=["website:start", "runtime:build", "website:build"] other=false
[DEBUG] moon_affected::affected_tracker  Task website:build is affected by  env=[] files=["website/blog/2024-10-01_moon-v1.29.mdx"] upstream=[] downstream=[] other=false

What marks an affected state is based on one or many of the following:

  • By touched files
  • By environment variables (task only)
  • By upstream dependencies
  • By downstream dependents (project only)
  • And other minor internal logic

This information is also included in the run report at .moon/cache/runReport.json, under the context.affected property. An example of this looks like:

{
  "projects": {
    "website": {
      "files": ["website/blog/2024-10-01_moon-v1.29.mdx"],
      "other": true
    },
    "runtime": {
      "downstream": ["website"],
      "other": true
    },
    "types": {
      "downstream": ["website", "runtime"],
      "other": true
    }
  },
  "tasks": {
    "website:build": {
      "files": ["website/blog/2024-10-01_moon-v1.29.mdx"],
      "other": false
    },
    "types:build": {
      "downstream": ["website:build"],
      "other": false
    },
    "runtime:build": {
      "downstream": ["website:build"],
      "other": false
    }
  }
}

Control upstream / downstream depth​

With this new tracker, we now have the ability to control the traversal depth for upstream dependencies and downstream dependents in moon query projects, via the --upstream and --downstream options respectively (the --dependents option is now deprecated).

These options support the following values:

  • none - Do not traverse deps.
  • direct - Traverse direct parent/child deps.
  • deep - Traverse full hierarchy deps.
$ moon query projects --affected --upstream none --downstream deep

What about tasks?​

We have the existing affected logic that has powered moon for years, and have updated that to include the new logging. However, it's not perfect and we want to improve it.

To support this overall enhancement for tasks, we need to support a task graph, which we currently do not. We only have a project graph (which has tasks), and an action graph (which has more than tasks). In a future release, we'll introduce a new task graph that will fill the gaps.

Experimental support for Pkl based configuration​

Pkl, what is that? If you haven't heard of Pkl yet, Pkl is a programmable configuration format by Apple. But what about YAML? YAML has served us well since the beginning, but we're not happy with YAML. It's better than JSON, TOML, and XML, but still has its downsides. We want something better, something that meets the following requirements:

  • Is easy to read and write.
  • Is dynamic and programmable (loops, variables, etc).
  • Has type-safety or built-in schema support.
  • Has Rust serde integration.

The primary requirement that we are hoping to achieve is adopting a configuration format that is programmable. We want something that has native support for variables, loops, conditions, and more, so that you could curate and compose your configuration very easily. Hacking this functionality into YAML is a terrible user experience in our opinion!

And with all that said, I'm sure you're curious what Pkl actually looks like in practice. Here's a few examples (unfortunately no syntax highlighting)!

type = "application"
language = "typescript"
dependsOn = List("client", "ui")

tasks {
  ["build"] {
    command = "docusaurus build"
    deps = List("^:build")
    outputs = List("build")
    options {
      interactive = true
      retryCount = 3
    }
  }
  ["typecheck"] {
    command = "tsc --build"
    inputs = new Listing {
      "@globs(sources)"
      "@globs(tests)"
      "tsconfig.json"
      "/tsconfig.options.json"
    }
  }
}

Pretty straight forward for the most part! Lists/Listings (arrays) are a bit different than what you may be used to, but they're super easy to learn.

Advanced examples​

I've talked a lot about programmable configs, but what exactly does that look like? Let's go through a few examples. Say you are building a Rust crate and you need a build task for each operating system. In YAML you would need to define each of these manually, but with Pkl, you can build it with a loop!

tasks {
  for (_os in List("linux", "macos", "windows")) {
    ["build-\(_os)"] {
      command = "cargo"
      args = List(
        "--target",
        if (_os == "linux") "x86_64-unknown-linux-gnu"
          else if (_os == "macos") "x86_64-apple-darwin"
          else "i686-pc-windows-msvc",
        "--verbose"
      )
      options {
        os = _os
      }
    }
  }
}

Or maybe you want to share inputs across multiple tasks. This can be achieved with local variables.

local _sharedInputs = List("src/**/*")

tasks {
  ["test"] {
    // ...
    inputs = List("tests/**/*") + _sharedInputs
  }
  ["lint"] {
    // ...
    inputs = List("**/*.graphql") + _sharedInputs
  }
}

Pretty awesome right? This is just a taste of what Pkl has to offer! We highly suggest reading the language reference, the standard library, or looking at our example configurations while testing Pkl.

In the future, if Pkl seems like the right fit, we plan to take full advantage of what it has to offer, by creating our own Pkl projects, modules, and types!

Caveats and restrictions​

Since this is an entirely new configuration format that is quite dynamic compared to YAML, there are some key differences to be aware of!

  • Each .pkl file is evaluated in isolation (loops are processed, variables assigned, etc). This means that task inheritance and file merging cannot extend or infer this native functionality.

  • default is a special feature in Pkl and cannot be used as a setting name. This only applies to template.yml, but can be worked around by using defaultValue instead.

template.yml
variables {
  ["age"] {
    type = "number"
    prompt = "Age?"
    defaultValue = 0
}
  • local is also a reserved word in Pkl. It can be worked around by escaping it with backticks, or you can simply use the preset setting instead.
tasks {
  ["example"] {
    `local` = true
    # Or
    preset = "server"
  }
}
  • Only files are supported. Cannot use or extend from URLs.

How to use Pkl?​

As mentioned in the heading, Pkl support is experimental, and is not enabled by default. If you're interested in trying out Pkl, you can with the following:

  • Install pkl onto PATH. Pkl uses a client-server communication model.
    • Can also be installed with proto: proto plugin add https://raw.githubusercontent.com/milesj/proto-plugins/refs/heads/master/pkl.toml
  • Use the .pkl file extension instead of .yml.
  • Pass the --experimentPklConfig CLI option, or set the MOON_EXPERIMENT_PKL_CONFIG environment variable.
$ moon check --all --experimentPklConfig
# Or
$ MOON_EXPERIMENT_PKL_CONFIG=true moon check --all

Pkl can be used alongside YAML with no issues! We'll merge, inherit, and compose as usual.

What about X instead?​

There are a handful of other interesting or popular programmable configurations out there, so why isn't moon experimenting with those? The answer is, we may! Just so long as they meet the requirements. With that said, we do have some opinions below:

  • Starlark/Skylark - On our list to evaluate.
  • Nickel, Jsonnet - On our list to evaluate, but not a fan of the JSON-like syntax.
  • Dhall - While this meets most of our requirements, the syntax isn't as readable or user-friendly as we'd like.
  • CUE - No Rust support, so unlikely. It also works quite differently than the other tools.
  • KCL - Nice syntax and meets the requirements, but no Rust support.

If there's another format you think we should investigate, drop us a line in Discord!

Looking for contributors!​

Are you a fan of moon (or proto)? Interested in learning Rust or writing more Rust? Want to contribute to an awesome project (we think so)? Well it just so happens that we are looking for active contributors!

We have a very long roadmap of features we would like to implement, but do not have enough time or resources to implement them in the timeframe we would like. These features range from very small (low hanging fruit) to very large (and quite complex).

If this sounds like something you may be interested in, post a message in Discord and let us know! Only a few hours a week commitment is good enough for us.

Other changes​

View the official release for a full list of changes.

  • Added a new task option, cacheLifetime, that controls how long a task will be cached for.
  • Added a new task merge strategy, preserve, that preserves the original inherited value.
  • Added a new setting vcs.hookFormat to .moon/workspace.yml, that can customize the shell/file format for hooks.
  • Updated task outputs to support token and environment variables.
  • Updated moon query projects to include the project description as a trailing value.
  • Updated moon query tasks to include the task type and platform, and the task description as a trailing value.