Elastic CI Stack for GCP

⚠️ Active Development Warning This project is under active development and is not yet ready for production use. APIs, configuration options, and module interfaces may change without notice. Use at your own risk and expect breaking changes.

The Elastic CI Stack for GCP gives you a private, autoscaling Buildkite Agent cluster running on Google Cloud Platform. Use it to run your builds on your own infrastructure, with complete control over security, networking, and costs.

This is a GCP implementation inspired by Buildkite's Elastic CI Stack for AWS, built with Terraform.

Quick Start

Prerequisites

• Terraform >= 1.0
• Buildkite Account
• GCP Account with a project
• gcloud CLI configured

Deploy the Stack

Create a main.tf file:

module "elastic-ci-stack-for-gcp" {
  source  = "buildkite/elastic-ci-stack-for-gcp/buildkite"
  version = "0.1.0"

  # Required
  project_id                  = "your-gcp-project"
  buildkite_organization_slug = "your-org-slug"
  buildkite_agent_token       = "YOUR_AGENT_TOKEN"

  # Stack configuration
  stack_name      = "my-buildkite-stack"
  buildkite_queue = "default"

  # Scaling configuration
  min_size = 0
  max_size = 10

  # Instance configuration
  machine_type = "e2-standard-4"
  region       = "us-central1"
}

Deploy:

terraform init
terraform plan
terraform apply

The module will create:

• VPC network with Cloud NAT
• IAM service accounts with appropriate permissions
• Managed instance group with Buildkite agents
• Cloud Function for autoscaling metrics
• Health checks and autoscaling based on queue depth

Architecture

The stack is organized into four modules:

• Networking - VPC, subnets, Cloud NAT, and firewall rules
• IAM - Service accounts and permissions for agents and metrics
• Compute - Instance groups, autoscaling, and agent configuration
• Buildkite Agent Metrics - Cloud Function for publishing queue metrics

Examples

See the examples/ directory for complete working examples:

• Networking Example - Standalone VPC setup
• IAM Example - Service account configuration
• Compute Example - Full agent deployment with autoscaling

Custom Images

The stack includes Packer templates for building custom VM images with pre-installed Buildkite agents and GCP monitoring tools.

Contributing

We welcome contributions! Please see Contributing Guidelines and Code of Conduct.

Related Projects

• Elastic CI Stack for AWS - The original AWS implementation
• Buildkite Agent - The Buildkite agent source code

License

MIT License - see LICENSE file for details.