Cache
moon's able to achieve high performance and blazing speeds by implementing a cache that's powered by
our own unique smart hashing layer. All cache is stored in .moon/cache
, relative from the
workspace root (be sure to git ignore this folder).
Hashing
Incremental builds are possible through a concept known as hashing, where in multiple sources are aggregated to generate a unique hash. In the context of moon, each time a target is ran we generate a hash, and if this hash already exists we abort early (cache hit), otherwise we continue the run (cache miss).
The tiniest change may trigger a different hash, for example, changing a line of code (when an input), or updating a package version, so don't worry if you see a lot of hashes.
Our smart hashing currently takes the following sources into account:
- Command (
command
) being ran and its arguments (args
). - Input sources (
inputs
). - Output targets (
outputs
). - Environment variables (
env
). - Dependencies between projects (
dependsOn
) and tasks (deps
). - For Node.js tasks:
- Node.js version.
package.json
dependencies (including development and peer).tsconfig.json
compiler options (when applicable).
Archiving & hydration
On top of our hashing layer, we have another concept known as archiving, where in we create a
tarball archive of a task's outputs and store it in .moon/cache/outputs
. These are akin to build
artifacts.
When we encounter a cache hit on a hash, we trigger a mechanism known as hydration, where we efficiently unpack an existing tarball archive into a task's outputs. This can be understood as a timeline, where every point in time will have its own hash + archive that moon can play back.
File structure
The following diagram outlines our cache folder structure and why each piece exists.
.moon/cache/
# Stores hash manifests of every ran task. Exists purely for debugging purposes.
hashes/
# Contents includes all sources used to generate the hash.
<hash>.json
# Stores `tar.gz` archives of a task's outputs based on its generated hash.
outputs/
<hash>.tar.gz
# State information about anything and everything within moon. Toolchain,
# dependencies, projects, running targets, etc.
states/
# Files at the root pertain to the entire workspace.
<state>.json
# Files for a project are nested within a folder by the project name.
<project>/
# Information about the project, its tasks, and its configs.
# Can be used at runtime by tasks that require this information.
runfile.json
<task>/
# Contents of the child process, including the exit code and
# unique hash that is referenced above.
lastRun.json
# Outputs of last run target.
stderr.log
stdout.log