Learn · Clusters, queues and targeting

The cluster model

Buildkite’s infrastructure model is three layers deep.

A cluster groups queues of agents together with the pipelines that use them. It is the isolation boundary: a team can be given its own cluster and manage its own agent pools without touching anyone else’s.

A queue is a named group of agents within a cluster. It is what pipeline steps target, and it is where capacity decisions actually live.

An agent is the process that connects your infrastructure to Buildkite, polls for work, runs it, and reports back.

Cluster: production
  ├─ Queue: linux-small     → 20 agents
  ├─ Queue: linux-large     → 4 agents
  └─ Queue: macos           → 6 agents

Assigning an agent to a queue

A self-hosted agent joins a queue through its tags:

buildkite-agent start --token "$AGENT_TOKEN" --tags "queue=linux-large"

The same can be set in the agent configuration file or through environment variables. If no queue is given, the agent joins the cluster’s default self-hosted queue.

An agent can only be assigned to one self-hosted queue within a cluster. This is worth internalising, because the older unclustered agents did allow multiple queue tags, and configuration copied from that era does not behave the way it used to.

Choosing queue boundaries

Queues cost nothing to create, so draw them where a real distinction exists:

• Platform — macOS builds cannot run on Linux agents, so they need their own queue whatever else you do.
• Machine size — a queue of large instances for the handful of jobs that need them, so they are not occupied by trivial work.
• Trust level — jobs that can reach production credentials on agents that are configured differently from the ones running pull-request builds.

That last one is the important one, and it comes back in the security module: a queue boundary is also a secrets boundary.