Learn · Dynamic pipelines

Uploading steps at build time

A pipeline file is a starting point, not the whole build. A running step can generate more steps and hand them back.

steps:
  - label: ":pipeline: Generate"
    command: .buildkite/generate.sh | buildkite-agent pipeline upload

.buildkite/generate.sh writes pipeline YAML to standard output. buildkite-agent pipeline upload reads YAML from standard input and adds those steps to the build that is currently running. They start immediately.

The script can be anything that emits YAML — a shell script, a Ruby program, a Go binary. It runs on an agent with your repository checked out, so it can look at the actual state of the code:

#!/usr/bin/env bash
set -euo pipefail

echo "steps:"
for service in $(git diff --name-only origin/main | cut -d/ -f1 | sort -u); do
  echo "  - label: \":hammer: $service\""
  echo "    command: make -C $service test"
done

That pipeline only tests what changed. On a monorepo, this is usually the single largest saving available.

Why upload rather than write a file

You might wonder why the generated YAML is piped to the agent instead of written to pipeline.yml. Because the build is already running — it read its configuration when it started. Uploading is how you add steps to a build in flight, and it can happen as many times as you like, from as many steps as you like.