GitHub Copilot CLI is not “autocomplete in a shell”. It’s an agentic assistant that can read your code, plan multi-step work, run commands (with your approval), edit files, drive Git and gh, talk to GitHub.com, and call out to MCP servers and custom agents. It uses the same agentic harness as the Copilot coding agent, so what you get in the terminal is a genuine collaborator rather than a fancy prompt box.

This post is a practical, hands-on guide. We’ll cover installation, the core interactive workflow, plan mode, permissions, MCP servers, custom agents, and then finish with a set of tips and tricks for advanced usage.

What is Copilot CLI (and what makes it different)?

At a high level, Copilot CLI gives you:

• Terminal-native agentic development — build, edit, debug, and refactor without leaving your shell.
• GitHub integration out of the box — the GitHub MCP server is pre-configured, so it can work with repos, issues, and pull requests using natural language, authenticated with your existing GitHub account.
• Two ways to drive it — an interactive session for conversational, iterative work, and a programmatic (-p) mode for scripting and automation.
• Full control by default — every action that could modify or execute files is previewed and requires your approval, unless you explicitly opt in to broader permissions.

It’s available on all Copilot plans (Free, Pro, Business, Enterprise). If you get Copilot through an organisation, an admin needs to have the Copilot CLI policy enabled. It runs on Linux, macOS, and Windows (via PowerShell 6+ or WSL).

A note on cost: each prompt you submit consumes from your premium request / AI credit allowance, and the amount varies by model. Keep an eye on this with /usage, which we’ll cover later.

Installing Copilot CLI

There are several installation paths — pick whichever fits your platform and tooling.

Install script (macOS and Linux):

curl -fsSL https://gh.io/copilot-install | bash

Homebrew (macOS and Linux):

brew install copilot-cli
# or the prerelease channel for the newest features
brew install copilot-cli@prerelease

WinGet (Windows):

winget install GitHub.Copilot

npm (all platforms):

npm install -g @github/copilot

Because GitHub is iterating on this tool very quickly (there have been hundreds of releases), it’s worth keeping your client up to date to get the latest features and fixes.

First launch and authentication

Launch the CLI from a folder containing code you want to work with:

copilot

On first launch you’ll be asked to confirm you trust the files in (and below) the current directory — more on why that matters in the security section. If you’re not logged in, run:

/login

and follow the on-screen instructions. You can also authenticate headlessly using a fine-grained Personal Access Token with the “Copilot Requests” permission, exposed via the GH_TOKEN or GITHUB_TOKEN environment variable — handy for CI and containers.

By default Copilot CLI uses Claude Sonnet 4.5. You can switch models (for example to GPT-5) at any time with the /model slash command.

The core workflow: interactive sessions

The interactive interface is where you’ll spend most of your time. Start a session with copilot, then just talk to it. A prompt can be a simple question or a request to do real work.

Example 1: Ask a question about your codebase

> Where is rate limiting implemented in this service, and what limits are applied?

Copilot will search the code, read the relevant files, and answer with references to the exact locations — without you having to grep through anything.

Example 2: Make a change

> Change the background-color of H1 headings to dark blue

Copilot finds the relevant CSS file, proposes the edit, and asks for your approval before writing it. This preview-then-approve loop is the heart of how the CLI keeps you in control.

Example 3: Investigate history

> Show me the last 5 changes made to CHANGELOG.md. Who changed it, when, and give a brief summary of each change.

Here it drives Git on your behalf and summarises the results in plain language.

Example 4: Build something from scratch

> Use create-next-app and Tailwind to scaffold a Next.js dashboard that pulls
  build metrics from the GitHub API — build success rate, average duration,
  failed builds, and test pass rate. When it's done, give me clear instructions
  to build, run, and view it.

This is where the agentic nature shines: Copilot plans the work, runs the scaffolding commands (with approval), writes the code across multiple files, and then hands you run instructions.

Referencing files with @

To pull a specific file into your prompt as context, use @ followed by the path:

> Explain @config/ci/ci-required-checks.yml
> Fix the bug in @src/app.js

As you type a path, matching files appear below the prompt box; use the arrow keys and Tab to complete it. You can also drag and drop files, or paste images (and PDFs) directly — useful for handing Copilot a screenshot of a stack trace or a design mock, on models that support image input.

Running shell commands directly

Prefix input with ! to run a shell command yourself without involving the model:

!git status
!npm test

This is great for quickly checking state mid-conversation without spending a request.

Plan mode: think before you build

For anything non-trivial, plan mode is your friend. Press Shift+Tab to cycle into it. In plan mode, Copilot analyses your request, asks clarifying questions to nail down scope and requirements, and produces a structured implementation plan before writing a single line of code.

> [plan mode] Add OAuth2 login with GitHub as a provider to this Express app,
  including session handling and a protected /profile route.

Copilot will come back with questions (“Which session store do you want — in-memory, Redis, or database?”) and then a step-by-step plan. You catch misunderstandings early, stay in control of complex multi-step tasks, and only proceed to implementation once the plan looks right. Press Shift+Tab again to cycle back out and execute.

Rolling back when things go wrong

Copilot doesn’t always do exactly what you had in mind — it might edit the wrong file, take a task in an unexpected direction, or run a command whose results you’d rather undo. When that happens, you can rewind the session to an earlier prompt. Press Esc twice in quick succession (with an empty input box), or use the /undo slash command — or its alias /rewind — to open the rewind picker.

The picker lists the rewind points for your session, most recent first, showing the start of each prompt and how long ago you submitted it. Choosing one takes you back to the state before Copilot started working on that prompt, and the selected prompt is dropped back into the input box so you can tweak and resubmit it.

There are two rewind behaviours, and Copilot picks the best one for your environment automatically:

• Git-based rewind rolls the whole workspace back to a snapshot taken at the start of the prompt. This is thorough but blunt: it reverts everything after that point — Copilot’s changes, your own manual edits, and the effects of shell commands — and deletes any files created since the snapshot. It requires a Git repository with at least one commit.
• Tools-based rewind is more surgical, letting you choose “Conversation only” (rewind the chat history but leave files untouched) or “Conversation + files” (also restore files Copilot changed). It’s currently experimental, so enable it first with /experimental on or the --experimental flag.

A word of caution: rewinding can’t be undone — once you roll back, the later session history is gone for good. Afterwards, it’s worth confirming the result with a quick shell command (remember the ! prefix), for example !git status, !git log --oneline -1, or !git diff, without leaving the CLI.

Understanding permissions (and staying safe)

By default, the first time Copilot wants to use a tool that could modify or execute files — touch, chmod, node, sed, and so on — it asks for approval. You’ll typically see three options:

1. Yes
2. Yes, and approve TOOL for the rest of the running session
3. No, and tell Copilot what to do differently (Esc)

• Option 1 allows this one command, this once.
• Option 2 allows that tool for the rest of the session — convenient for something like chmod you’ll use repeatedly, but be careful: approving rm this way lets Copilot run any rm (including rm -rf ./*) without asking again.
• Option 3 cancels and lets you steer it in a different direction. You can give inline feedback on the rejection so Copilot adapts without stopping entirely.

Fine-grained control from the command line

You can pre-authorise or block specific tools using flags that work in both interactive and programmatic modes:

# Allow a specific tool without prompting
copilot --allow-tool='shell(git)'

# Allow all shell commands, but explicitly block the dangerous ones
copilot --allow-all-tools --deny-tool='shell(rm)' --deny-tool='shell(git push)'

# Allow all tools from an MCP server except one specific tool
copilot --allow-tool='My-MCP-Server' --deny-tool='My-MCP-Server(tool_name)'

A few things worth knowing:

• --deny-tool always takes precedence over --allow-tool and --allow-all-tools.
• For git and gh, you can scope to a subcommand, e.g. shell(git push).
• --allow-tool='write' lets Copilot edit files without per-file approval.
• Inside a session, /allow-all and /yolo (or launching with --allow-all / --yolo) enable everything at once.

Use the broad options with care. With --allow-all-tools, Copilot has the same access to your machine that you do and can run any command without prior approval.

Trusted directories and sandboxing

Copilot CLI only operates within directories you’ve marked as trusted, which is why it prompts you on launch. Don’t launch it from your home directory or from folders full of untrusted executables or sensitive data. You can add directories mid-session with /add-dir /path/to/dir, and switch the working directory with /cwd or /cd.

For extra safety when using automatic approvals, run inside a sandbox:

• Local sandboxing — run /sandbox enable inside a session to restrict Copilot’s access to your filesystem, network, and system capabilities.
• Cloud sandboxing — start an isolated, cloud-hosted session with copilot --cloud. This keeps session state between uses, lets you continue from a different machine, and run tasks in parallel — without touching your local environment. (Sandboxes are in public preview.)

Working with GitHub.com

Because the GitHub MCP server is pre-configured, Copilot CLI is fluent in your GitHub workflow. Some things you can just ask for:

> List my open PRs
> List all open issues assigned to me in OWNER/REPO
> I've been assigned https://github.com/octo-org/octo-repo/issues/1234 —
  start working on it for me in a suitably named branch.
> Create a PR that changes the "How to run" heading to "Example usage" in the
  README of https://github.com/octo-org/octo-repo
> Check the changes in PR https://github.com/octo-org/octo-repo/pull/57575 and
  report any serious errors.
> Merge all of the open PRs I've created in octo-org/octo-repo

When Copilot opens a PR or issue, it does so on your behalf, and you’re recorded as the author. This turns the CLI into a genuinely useful “assign it and walk away” tool for small, well-scoped changes.

Programmatic mode: Copilot in scripts and pipelines

Pass a single prompt with -p (or --prompt) and the CLI completes the task and exits — perfect for automation. Pair it with approval flags so it can run unattended:

copilot -p "Show me this week's commits and summarise them" --allow-tool='shell(git)'

copilot -p "Revert the last commit, leaving the changes unstaged" --allow-all-tools

You can also generate flags dynamically and pipe them in:

./script-outputting-options.sh | copilot

This unlocks patterns like nightly summarisation jobs, scripted refactors across many repos, or a CI step that asks Copilot to review a diff. Just remember: with automatic approval, there’s no human in the loop, so sandboxing and tightly-scoped --deny-tool rules become important.

Working autonomously: autopilot and /delegate

Sometimes you don’t want to babysit each step — you just want to hand off a task. Copilot CLI gives you two ways to do this, and the difference comes down to where the work happens.

Autopilot mode runs locally. You grant full permissions and Copilot works through the task without stopping to prompt you, while you watch progress in real time on your own machine. Enable it interactively by pressing Shift+Tab until you see “autopilot” in the status bar, then enter your prompt. Or run it programmatically:

copilot --autopilot --yolo --max-autopilot-continues 10 -p "Migrate the test suite from Jest to Vitest and make it pass"

The --max-autopilot-continues flag is a useful safety valve — it caps how many times autopilot will keep going before handing back control.

/delegate pushes the work to the cloud. Instead of running locally, it hands the task to the Copilot cloud agent on GitHub: Copilot creates a branch, opens a draft pull request, and works in the background — so the task keeps running even if you close your laptop.

/delegate complete the API integration tests and fix any failing edge cases

You can also prefix a prompt with & as shorthand for the same thing:

& complete the API integration tests and fix any failing edge cases

Copilot will offer to commit any unstaged changes as a checkpoint on the new branch, then give you a link to the pull request and cloud agent session so you can follow along and review when it’s done. Reach for autopilot when you want hands-free local execution, and /delegate when you want to offload a task entirely and get on with something else.

Running work in parallel with /fleet

When a task breaks down into several independent pieces, the /fleet slash command speeds things up by farming the work out to multiple subagents that run in parallel. It pairs naturally with plan mode: build a plan first, then let a fleet execute it.

The typical flow is:

1. Press Shift+Tab to switch into plan mode and describe the feature or change you want.
2. Work with Copilot to refine the implementation plan.
3. Once the plan looks right, either choose “Accept plan and build on autopilot + /fleet” to have it work autonomously with subagents, or exit plan mode and prompt it yourself:

/fleet implement the plan

Copilot then splits the work across subagents, running independent parts concurrently. Monitor them with the /tasks slash command, which lists the background tasks for the session — including each subagent’s subtask. Use the arrow keys to navigate; press Enter to view details (or a summary once complete), k to kill a process, and r to clear finished ones. Press Esc to return to the prompt.

Extending Copilot CLI

This is where the CLI goes from “useful” to “tailored to how your team works”.

MCP servers

Model Context Protocol (MCP) servers give Copilot access to additional data sources and tools. The GitHub server ships by default; to add more:

/mcp add

Fill in the details (use Tab to move between fields) and press Ctrl+S to save. Configurations live in ~/.copilot/mcp-config.json (override the location with COPILOT_HOME). You can list configured servers and their tool names with /mcp — useful when you want to reference a server precisely in an --allow-tool / --deny-tool rule.

Tip: if you know a specific MCP server can do the job, name it in your prompt — e.g. “Use the GitHub MCP server to find good first issues in octo-org/octo-repo” — to steer Copilot toward the result you want.

Custom instructions

Copilot CLI automatically reads custom instructions from your repository, so it understands your conventions and how to build, test, and validate changes. It supports:

• Repository-wide: .github/copilot-instructions.md
• Path-specific: .github/instructions/**/*.instructions.md
• Agent files: AGENTS.md

All applicable instruction files now combine rather than falling back by priority, so you can layer general and path-specific guidance.

Tip: keep your instructions files minimal for efficient token usage. They’re injected into every prompt, so every line costs context. Include only the essential guidance for working in the repository — build and test commands, key conventions, and anything non-obvious — and leave out anything Copilot can readily infer from the code itself.

Custom agents

Custom agents are specialised versions of Copilot for particular workflows. The CLI ships with a set of built-in ones it can delegate to automatically:

You can invoke agents three ways:

# Pick from a list interactively
/agent

# Ask for one in natural language
> Use the code review agent to check my staged changes

# Select one at launch
copilot --agent=refactor-agent --prompt "Refactor this code block"

Define your own agents as Markdown “agent profiles” specifying their expertise, allowed tools, and instructions — at user level (~/.copilot/agents), repository level (.github/agents), or organisation/enterprise level (.github-private/agents). On naming conflicts, user overrides repository overrides organisation.

Skills and hooks

Two more extension points let you package know-how and enforce guardrails.

Skills enhance Copilot’s ability to perform specialised tasks with bundled instructions, scripts, and resources. A skill is a folder containing a SKILL.md file (with a name and description) plus any helper scripts or reference material. Copilot reads the description to decide when the skill is relevant, then follows its instructions.

For example, a skill that standardises how release notes are generated might live at .github/skills/release-notes/SKILL.md:

---
name: release-notes
description: Generate release notes from merged PRs since the last tag, grouped
  by Added / Changed / Fixed, following our CHANGELOG format.
---

# Release notes

1. Find the most recent git tag.
2. Collect merged PRs since that tag using `gh pr list --state merged`.
3. Group them by Conventional Commit type into Added / Changed / Fixed.
4. Write the result to the top of `CHANGELOG.md`, keeping the existing entries.

Now a prompt like “Draft the release notes for the next version” will pick up the skill and produce output that matches your conventions every time.

Hooks let you run custom shell commands automatically at key points in a session — great for validation, logging, security scanning, or workflow automation. Because a hook fires deterministically around agent events, it’s a good place to enforce guardrails rather than relying on the model to remember a rule.

Hooks are configured in JSON. Create a NAME.json file in .github/hooks/ (repository-level) or ~/.copilot/hooks/ (user-level), with a version and a hooks object keyed by event — the available triggers include sessionStart, sessionEnd, userPromptSubmitted, preToolUse, postToolUse, and errorOccurred.

Each entry provides both a bash command (for Linux/macOS) and a powershell command (for Windows), so the hook runs everywhere. For example, a preToolUse hook that logs every tool the agent is about to run — handy for auditing — would live in .github/hooks/audit.json:

{
  "version": 1,
  "hooks": {
    "preToolUse": [
      {
        "type": "command",
        "bash": "echo \"$(date): about to run tool\" >> logs/agent-audit.log",
        "powershell": "Add-Content -Path logs/agent-audit.log -Value \"$(Get-Date): about to run tool\"",
        "cwd": ".",
        "timeoutSec": 10
      }
    ]
  }
}

You can also point a hook at an external script instead of an inline command — useful for anything non-trivial, like scanning a proposed change for secrets before it’s written:

{
  "version": 1,
  "hooks": {
    "preToolUse": [
      {
        "type": "command",
        "bash": "./scripts/scan-secrets.sh",
        "powershell": "./scripts/scan-secrets.ps1",
        "cwd": "scripts",
        "timeoutSec": 30
      }
    ]
  }
}

The hook receives details of the event as JSON on stdin (timestamp, working directory, tool name, and arguments), so your script can inspect what’s happening and act accordingly. A couple of things to remember: hook config is loaded when the CLI starts, so restart after editing it; any referenced script needs to be executable (chmod +x) with a proper shebang; and the default timeout is 30 seconds, tunable with timeoutSec.

Copilot Memory

Normally, anything you teach Copilot in a session is forgotten when that session ends — so the next time you start up, you’re explaining your conventions all over again. Copilot Memory fixes that by letting the agent keep notes about your repository between sessions.

As it works, Copilot notices recurring facts, conventions, and preferences and saves them as short “memories”. For instance, if you correct it once — “we use Vitest here, not Jest”, or “always put new API routes under src/routes/ and export them from the barrel file” — it can store that and apply it automatically next time, without you having to repeat yourself.

The practical payoff is that later sessions get progressively more useful: Copilot already knows how your project is laid out, which tools you use, and how you like things done, so you spend less time on context and more on the actual task. Think of it as the agent gradually onboarding itself to your codebase.

Tips and tricks for advanced usage

Here’s the stuff that makes daily use noticeably better.

Review before you commit. Run /review to have Copilot analyse your uncommitted changes and give you feedback right in the terminal — no PR required. You can narrow the scope by adding a prompt, path, or file pattern after the command (e.g. /review focus on error handling in src/api). If Copilot needs to run a command to inspect the diff, it’ll ask first; then it reports back any issues so you can fix them before they ever reach a pull request.

Steer while it’s thinking. You don’t have to wait for Copilot to finish. Enqueue follow-up messages to redirect it or queue up the next instruction, and it’ll feel far more like a real conversation.

Connect it to VS Code for the best of both worlds. If you like the speed of the terminal but miss the visual tools of an editor, connect the CLI to VS Code. Start Copilot from a directory that matches an open (trusted) VS Code workspace and it connects automatically — you’ll see “Visual Studio Code connected” in the startup message — or use the /ide slash command to connect, switch, or disconnect at any time. This works whether you’re in VS Code’s integrated terminal or an external one. Once connected you get: your editor selection as context (highlight code and just say “Debug this” — no file paths needed), proposed edits shown as side-by-side diffs you accept or reject visually, access to VS Code’s live diagnostics so Copilot can fix errors your editor already spotted, and the ability to view CLI session transcripts in the Copilot Chat Sessions view and Resume in Terminal to pick up where you left off.

Stop instantly. Pressed enter and immediately regretted the prompt? Hit Esc while it’s “Thinking” to stop.

Resume where you left off. Sessions are saved with their context:

copilot --continue          # resume the most recently closed session
copilot --resume            # pick a session to resume

You can even kick off a Copilot cloud agent session on GitHub and pull it down into your local CLI to keep going.

Manage your context window. Long sessions are handled automatically — Copilot auto-compacts your history in the background at ~95% of the token limit, enabling effectively endless sessions. To take manual control:

• /context — a visual breakdown of current token usage
• /compact — compress history now (press Esc to cancel)
• /usage — AI credits used, session duration, lines edited, and per-model token breakdown

Schedule prompts. Automate recurring or delayed work right from a session:

/every 1h Run frontend tests and report any failures
/after 30m Summarise today's merged PRs

At time of writing, scheduled prompts are currently an experimental feature, so you’ll need to enable experimental mode first — either with the /experimental on slash command inside a session, or by launching with the --experimental command-line option.

Toggle reasoning visibility. Press Ctrl+T to show or hide the model’s reasoning as it works. The setting persists across sessions — handy when you want to understand how it reached a conclusion.

Tune settings without leaving the CLI. Use /settings to open an interactive editor, /settings KEY VALUE to set something directly (works in scripts too), and /settings show KEY to print a value. Ctrl+R restores a setting to its default.

Pick the right model for the job. Newer models support a 1M-token context window and configurable reasoning levels. Larger context and higher reasoning consume more credits, so use the regular settings by default and reach for the big context / deep reasoning only on genuinely complex, multi-file tasks.

Bring your own model. You can point Copilot CLI at an OpenAI-compatible endpoint, Azure OpenAI, Anthropic, or a local model like Ollama, using environment variables (COPILOT_PROVIDER_BASE_URL, COPILOT_PROVIDER_TYPE, COPILOT_PROVIDER_API_KEY, COPILOT_MODEL). The model must support tool calling and streaming; run copilot help providers for details.

Wire it into other tools via ACP. Copilot CLI can act as an agent over the Agent Client Protocol (ACP), so IDEs and automation systems that speak ACP can use it directly. See Copilot CLI ACP server for more info.

Voice input. On supported setups you can speak your prompt instead of typing — handy when your hands are busy or you’re describing something long. First, run /voice in a session to download the local speech runtime and a voice model (transcription runs entirely on your machine — no audio leaves it; English and Spanish are supported). Then dictate one of two ways: hold the space bar while you talk and release when done for short prompts, or press Ctrl+X then V to toggle recording on for longer ones (press any key to stop). The transcribed text lands at your cursor so you can tweak it before submitting.

Discover everything fast. Type ? in a session, or run copilot help in your terminal. For deeper reference, use the subcommands copilot help config, copilot help environment, copilot help logging, and copilot help permissions.

A pragmatic workflow to adopt

If you want a starting point, this is a solid loop:

1. Launch copilot from your project root and trust the directory.
2. For anything beyond a one-liner, drop into plan mode (Shift+Tab) and refine the plan before any code is written.
3. Let it work, approving tools individually until you trust the pattern, then approve for the session.
4. Use ! for quick checks (!git diff, !npm test) without spending requests.
5. Run /review (or ask the code review agent) to sanity-check the diff before you commit.
6. Have Copilot commit, push, and open a PR — then keep an eye on /usage.

For unattended automation, script it with -p, scope permissions tightly with --allow-tool / --deny-tool, and run inside a sandbox.

Wrapping up

GitHub Copilot CLI brings a real agentic collaborator to the place developers already work. The core loop — converse, plan, approve, execute — keeps you firmly in control while offloading the mechanical parts of building, debugging, and shipping. Layer on MCP servers, custom agents, custom instructions, and memory, and it starts to feel like a teammate that already knows your codebase and conventions.

The tool is evolving quickly, so keep your client updated, start small with tightly-scoped tasks, and lean on plan mode and sandboxes as you build trust. Once it’s part of your muscle memory, you’ll wonder how you shipped without it.

Will Larson’s new book, Crafting Engineering Strategy: How Thoughtful Decisions Solve Complex Problems, sets out to dismantle that complaint. Larson is the author of An Elegant Puzzle, Staff Engineer and The Engineering Executive’s Primer, and here he turns his attention to the one topic engineers most love to grumble about and least understand how to influence. His central claim is refreshingly unglamorous. Strategy, he writes, is “the art of reproducibly making good decisions.” Not visionary pronouncements from on high, not a glossy deck presented at the all-hands, but the boring, repeatable practice of deciding well and writing it down.

I found this to be one of the most immediately useful engineering books I’ve read in years, precisely because it strips away the mystique. This post walks through the ideas that stuck with me, and then looks at how the book can be applied by different roles across an engineering organisation, because one of Larson’s most encouraging arguments is that strategy is not the exclusive property of executives.

There is always a strategy

The book opens by refusing to accept the premise that a strategy is missing. Larson has “never worked somewhere where people didn’t claim there was no strategy,” and yet he’s also never found a company that genuinely lacked one. His point is that a strategy is embedded in the decisions an organisation actually makes, whether or not anyone has bothered to write it down. Borrowing William Gibson’s famous line about the future being unevenly distributed, he suggests the strategy is already there; it’s just visible only to a small group, and often quickly forgotten.

This reframing matters because it changes the question. Instead of asking “why don’t we have a strategy?”, Larson wants you to ask where the strategy lives if you can’t find it. When you go looking, you’ll usually discover an implicit set of rules that nobody has examined and nobody can therefore improve. His prescription is disarmingly simple, and it’s the thread running through the entire book:

The single biggest act you can take to further strategy in your organization is to write down strategy so it can be debated, agreed upon, and explicitly evolved.

Writing it down is what makes it possible to precisely disagree. He describes his time at Carta, where a written strategy meant new joiners could argue with decisions “from an informed place rather than coming across as attached to their prior company’s practices.” Oral history gives you a version of events that depends entirely on who you happened to ask. Written history lets an organisation agree at scale, which he argues is the prerequisite for growing at scale rather than trapping all the context inside a handful of senior heads.

The five steps

The backbone of the book is a repeatable, five-step structure for producing a strategy: explore, diagnose, refine, set policy, and operate. Larson is candid that this isn’t a magic formula, and that “strategies fail more often due to avoidable errors than from fundamentally unsound thinking. Busy people skip steps. Especially steps they dislike or have failed at before.” The structure exists to stop you skipping the awkward parts.

Exploration comes first, and it’s deliberately judgement-free. Before committing to an approach, you look at how other teams and companies have handled the same problem. His example is Uber’s 2014 service migration, where the team grounded themselves by reading the Google Borg and Berkeley Mesos papers before deciding anything. The discipline here is resisting the urge to reach for whatever worked at your last job.

Diagnosis is where the book earns its keep, and Larson calls it strategy’s foundation. This is the deliberate act of understanding your problem before reaching for a solution. He is blunt about the stakes:

Every strategy that I’ve seen fail, did so due to a lazy or inaccurate diagnoses. It’s very challenging to fail with a proper diagnosis, and almost impossible to succeed without one.

A good diagnosis, he argues, isn’t a statement of opinion you can easily dispute; it’s “a web of interconnected observations, facts, and data” that is hard to argue against even for people who dislike the conclusions. Two ideas from this chapter are especially worth highlighting. The first is that you should actively seek out the people most likely to disagree with you and represent their view faithfully, without letting your own bias show through. The second is his instruction to reframe blockers as part of the diagnosis rather than as reasons to give up. If your executives keep changing their minds, that isn’t proof that strategy is impossible; it’s a condition your strategy has to account for. As he puts it, “there are never reasons why strategy simply cannot succeed, only diagnoses you’ve failed to recognize.”

He’s also honest about organisational politics. Sometimes the true diagnosis is that a particular leader is the problem, and you cannot write that down. His advice is to “whisper the controversial parts”, finding a professional, factual framing. His example from a private equity transition simply notes that new ownership “will expect us to reduce R&D headcount costs” without editorialising about how anyone feels about it.

Refinement is the step most people skip. This is where you pressure test a raw diagnosis using one of three techniques: strategy testing, systems modelling, and Wardley mapping. Strategy testing is his antidote to what he calls the waterfall strategy trap: rather than perfecting a document in a vacuum, you run your proto-strategy against reality and iterate. His deeper point is that the biggest risk isn’t modelling too early or mapping too late, it’s doing neither, so he keeps the barrier to entry low.

Policy is where decisions finally get made. “Policy is interpreting your diagnosis into a concrete plan,” and he divides policies into four recurring kinds: approvals, allocations, direction, and guidance. Direction is unambiguous instruction, as in Calm’s blunt “all new code must be written in the monolith”, which removed a debate that individual judgement kept reopening. Guidance is for when you know the destination but can’t mandate the route. Allocations are the most concrete statement of priority an organisation makes; Uber’s decision to fix “one full time engineer on manual service provisioning tasks” and pour everyone else into automation is a policy that says more about priorities than any mission statement could. He offers two criteria for judging a policy: it must be applicable to real, messy situations, and it must be enforced. A policy such as “we only hire world-class engineers” fails both, because nobody can define or consistently apply it.

Operations is the step that separates strategies that change behaviour from strategies that “fade quietly into your organization’s history.” Policies don’t implement themselves. Larson’s memorable framing is that “effectively operating a policy is two thirds avoiding common practices that simply don’t work” such as top-down pronouncements, one-off training, and the vague hope that you’ll “just change the culture.” The remaining third is choosing concrete mechanisms: approval forums, inspection, automation, and above all nudges. He’s a strong advocate for the humble nudge, recalling how Stripe told teams when their cloud spend accelerated rather than forbidding all new spend outright, and concluding that “done well, I continue to believe they are the most effective operational mechanism.”

Putting it together: a worked example

Laid out one after another, the five steps can still feel abstract, so it’s worth walking a single problem all the way through them. The book leans on real case studies from Uber, Stripe and Calm to do this; what follows is my own illustrative scenario rather than one of Larson’s, but it’s the kind of situation many scale-ups will recognise.

Imagine a company that has grown from thirty engineers to a hundred and fifty in two years. New services are appearing faster than anyone can count, on-call rotas are miserable, and the small platform team is drowning in provisioning requests. Everyone agrees “something should be done”, but every attempt to fix it dissolves into an argument about microservices versus monoliths.

Exploration comes first, and it stays deliberately opinion-free. You read how comparable companies handled service sprawl, you dig out the internal design docs behind the three services that caused the most pain last year, and you note which patterns keep recurring. You resist the urge to announce a conclusion. The goal is simply to understand the shape of the problem space before committing to anything.

Diagnosis turns that reading into an honest, evidence-backed picture of your specific situation. Perhaps the data shows that the platform team of four hasn’t grown despite the engineering organisation tripling, that ninety per cent of new services were created to dodge a slow provisioning queue rather than for any architectural reason, and that most services have fewer than two committers. Following Larson’s advice, you deliberately sit down with the staff engineer who is convinced microservices are the future, and you capture their view faithfully. You also reframe the obvious blocker, “the platform team is understaffed and that won’t change this year”, not as a reason to give up but as a constraint the strategy must be built around. That single move, treating the immovable object as part of the diagnosis, is what stops the whole effort stalling.

Refinement pressure tests that picture before you commit. A quick systems model might reveal that adding services actually slows the organisation down once the platform team saturates, which is a far more persuasive argument than any assertion. Rather than perfecting a grand document, you run a small strategy test: for one month, route all new service requests through a lightweight review and see what actually happens. You learn cheaply, and you gather real evidence instead of speculation.

Policy is where the decisions finally land, and the diagnosis makes them feel almost inevitable. You might set a piece of direction (“new functionality is built in the existing service that owns that domain unless an exception is granted”), an allocation that fixes one platform engineer on manual provisioning while the rest build self-service tooling, and some guidance encouraging teams to fold thinly-staffed services back into their owning domain where it makes sense. Each policy maps to a specific line of the diagnosis, and each one is both applicable and enforceable, which is Larson’s bar for a policy worth having.

Operations is what makes any of that real. Instead of a launch email and a hopeful shrug, you add a nudge that pings a team the moment they try to create a new service, pointing them at the guidance and the exception process. You stand up a fortnightly inspection of new services and provisioning lead times on a shared dashboard that “cannot silently fail”, and you route exception requests through a single lightweight approval channel. None of this depends on a heroic mandate; it depends on mechanisms that keep working after the initial announcement has been forgotten.

Run end to end, the example shows why Larson insists the awkward early steps matter most. By the time you reach policy, the hard thinking is done and the decisions write themselves. Skip exploration and diagnosis, and you’d have jumped straight to “stop making so many microservices”, a decree that sounds decisive and changes nothing.

How different roles can apply the book

The reason I’d recommend this book widely is that it explicitly rejects the idea that you need a title before you’re allowed to think strategically. “You can work on strategy from anywhere in an organization,” Larson writes. “It just requires different tactics to do so.” Here’s how that plays out across the roles you’ll find in most engineering organisations.

Engineers

If you’re an individual contributor without formal authority, the book’s most valuable gift is a pair of low authority techniques. The first, borrowed from Staff Engineer, is “take five, then synthesize”: document how five recent, related decisions were actually made, then synthesise them into a diagnosis and policy. You’re not inventing a strategy so much as naming the one that already exists, which makes it very hard for anyone to argue you’re overstepping. The second, from An Elegant Puzzle, is “model, document, and share”: adopt the approach yourself, write down your reasoning, and let people copy your success. Neither requires anyone’s permission. For an engineer who has ever felt powerless watching a bad decision unfold on a team they don’t own, this is a practical route to influence.

Staff and principal engineers

Senior individual contributors sit at exactly the altitude the book is aimed at. Larson introduces the concept of strategy altitude, being deliberate about the level at which a policy should be set, and Staff-plus engineers are often the people who can see when a decision is being made at the wrong one. They’re also central to his idea of informational herd immunity. You don’t need every engineer to have read the strategy; you just need “enough people who know something that confusion doesn’t propagate too far.” In practice, that means every Staff-plus engineer and manager understanding the strategy well enough to keep their team on track. The refinement toolkit, particularly systems modelling and Wardley mapping, is squarely in this group’s wheelhouse and gives them a rigorous way to challenge or support a proposed direction with evidence rather than opinion.

Engineering managers

Managers are the connective tissue that keeps a written strategy alive, and the book’s chapter on operations reads almost like a manual for them. Larson is scathing about the mechanisms managers most often reach for by default, warning that “a couple of trainings will never change organizational behavior.” The more effective path is the unglamorous one: inspection mechanisms that “cannot silently fail”, nudges that bring context to people at the moment they need it, and a genuine curiosity about why some teams aren’t adopting a change rather than assuming they’re being difficult. His observation that people usually “want to do things the new way, but rarely take time to learn how to do it” is a healthier default assumption than most performance conversations start from.

Engineering executives

Executives get the book’s most bracing warning. Yes, they can mandate adherence in a way engineers can’t, but “mandates only matter if there are consequences”, and an executive unwilling to enforce them is issuing empty instructions. More pointedly, Larson notes that access and mandates “often create the appearance of progress” without improving the underlying diagnosis, which is “why executive strategies can fail so spectacularly and endure so long despite failure.” His conclusion is one every senior leader should sit with: executives “have an easier time doing strategy, but a much harder time learning how to do strategy well.” That’s the argument for practising the craft long before you reach the top, because “waiting to do strategy until you are an executive is a recipe for disaster.”

Product managers and other partners

Even outside engineering, the book insists you can contribute, though it will demand an even more influence driven approach. The examples where engineers themselves are on uncertain ground, such as the early adoption of large language models, are exactly the moments when an outside perspective is most valuable. The through-line is the same for everyone: diagnose honestly, write it down, and let the artefact do the persuading over time.

The good, and the caveats

What I appreciated most is how grounded the book is in real, named examples. The case studies aren’t sanitised. Larson revisits Digg’s catastrophic V4 rewrite, which he calls “the worst considered strategy I’ve personally participated in” and which “ensured we died fast rather than having an opportunity to dig our way out.” He’s careful, though, to relabel such strategies “inappropriate” rather than “bad”, because “the same strategy might well be very effective in a different set of circumstances.” That intellectual honesty runs throughout, and it’s a welcome corrective to the confident, context-free advice that dominates so much engineering leadership writing.

The book is not a light read. The five step scaffolding, the refinement techniques and the taxonomy of policies reward careful study rather than a quick skim, and readers hoping for a checklist to copy will be gently disappointed. That’s by design; Larson repeatedly reminds you that “the structure is not sacrosanct, it’s the thinking behind the sections that really matter.” The chapters on systems modelling and Wardley mapping in particular are enough to get you started but will send the curious off to other sources. And if you’ve read his earlier work, you’ll recognise a fair amount of the philosophy, albeit sharpened and far better organised here.

Crafting Engineering Strategy delivers on its subtitle by treating strategy as a learnable, teachable craft rather than an innate gift, and it does so with enough concrete examples that you can start applying it the same day. If you’ve ever complained that your organisation lacks a strategy, Larson’s response is both a challenge and an invitation: it’s already there, so go and write it down. As he reminds us, strategy is useful, and doing strategy can be simple, too.

You can find the book on Amazon or via O’Reilly, and much of the material is available to read on Larson’s companion site.

This post documents the symptoms, the root cause, and the exact steps required to recover a cluster stuck in this half‑installed, half‑removed state.

The Setup

This cluster was:

• Running AKS with the old AAD Pod Identity Helm chart still installed
• Missing MIC (Managed Identity Controller)
• Still running NMI (Node Managed Identity) on every node
• Running Flux v2 with image automation enabled
• Using ACR with a node‑assigned managed identity

This combination is more common than you’d think—especially in clusters upgraded over several years.

The Symptoms

Flux’s image-reflector-controller began failing with IMDS authentication errors:

failed to configure authentication options: failed to create provider access token for the controller: ManagedIdentityCredential: context deadline exceeded

NMI logs showed:

failed to get matching identities for pod: flux-system/image-reflector-controller…

Flux could not:

• Query ACR
• Populate its tag database
• Resolve image policies

Image automation was effectively dead.

Diagnosis: The Smoking Gun

Running kubectl get pods -n kube-system | grep nmi revealed dozens of NMI pods, all running for hundreds of days.

kubectl get crd | grep aadpodidentity revealed AAD Pod Identity CRDs still installed.

But running kubectl get pods -n kube-system | grep mic showed no MIC pods at all.

This is the broken state:

• NMI intercepts IMDS calls
• MIC is missing, so identities are never assigned
• Every IMDS call from a pod fails
• Flux cannot authenticate to ACR

This is where the cluster was stuck.

Why This Breaks Flux

Flux’s controllers authenticate to ACR using the node’s managed identity via IMDS.

But NMI intercepts IMDS calls and tries to look up a pod’s assigned identity.

With MIC missing, NMI always returns:

no AzureAssignedIdentity found

Flux never reaches IMDS → never reaches ACR → never sees image tags.

The Fix: AzurePodIdentityException

The solution is to tell NMI:

“Do not intercept IMDS calls for Flux pods.”

This is done using an AzurePodIdentityException.

But here’s the twist:
AAD Pod Identity has multiple CRD schemas depending on the version, and this cluster was running a very old one.

Attempts using modern fields like:

spec:
  podSelector:

or

spec:
  PodLabels:

were rejected.

The correct schema, discovered by inspecting existing exceptions (kubectl get azurepodidentityexception -n flux-system -o yaml), was:

spec:
  podLabels:
    <label>: <value>

I discovered a couple of existing exception for Flux that didn’t cover the label used by the image-reflector-controller pod:

spec:
  podLabels:
    app.kubernetes.io/name: flux-extension

Creating the Correct Exception

I inspected the labels of the pods in the flux-system namespace:

kubectl get pods -n flux-system --show-labels

Flux controllers all share this label:

app.kubernetes.io/name=microsoft.flux

So the correct exception is:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzurePodIdentityException
metadata:
  name: flux-system-exception
  namespace: flux-system
spec:
  podLabels:
    app.kubernetes.io/name: microsoft.flux

Apply it:

kubectl apply -f flux-system-exception.yaml

Then restart Flux:

kubectl rollout restart deploy -n flux-system

Verifying the Fix

Check NMI logs:

kubectl logs -n kube-system -l app.kubernetes.io/component=nmi --tail=200

Before the fix:

failed to get matching identities for pod flux-system/image-reflector-controller…

After the fix no errors are shown.

Flux logs then showed healthy behaviour:

successful scan: found 1 tags

Latest image tag … resolved to prod-8552309f0-20260512T090128

Image automation was back.

Final Thoughts

This issue is obscure, poorly documented, and easy to miss. If you’re running Flux on AKS and see IMDS or ACR authentication failures, check for:

• NMI pods still running
• MIC missing
• Old AAD Pod Identity CRDs
• Missing or incorrect AzurePodIdentityException objects

Hopefully this post saves someone else the hours of digging it took to unravel this.

This post is a practical reference for:

• Signing OCI container images with cosign (Sigstore)
• Generating and attaching attestations (including SBOM-related examples)
• Storing signatures/attestations alongside images in Azure Container Registry (ACR)
• Understanding how the OCI Referrers API organises “artifacts about artifacts”
• Briefly comparing Cosign to Notation (Notary Project)

Key concepts (what are we actually storing?)

When people say “sign the image” they often mean a few related, but distinct things.

Signatures

A signature is a statement like:

“I (an identity) signed this immutable image digest.”

The important part is that you sign the image digest (@sha256:...), not the tag (:latest), because tags are mutable.

Attestations

An attestation is a signed statement about the image (or about another artifact attached to the image). Examples:

• Provenance: where the build ran, what sources were used, what inputs were consumed
• SBOM: a machine-readable list of packages/components contained in the image
• Vulnerability scan results (careful: these go stale quickly)

Cosign signs attestations using DSSE envelopes and commonly uses the in-toto attestation format.

SBOMs

An SBOM is a document — it’s valuable when you:

• Generate it consistently
• Store it alongside the exact digest you shipped
• Sign it (so people can trust its origin)
• Make it discoverable by automation

Why sign images and generate attestations?

If you’re already scanning images, signatures and attestations provide the missing chain of custody.

• Integrity: detect tampering (a signature ties to an immutable digest)
• Authenticity: verify “this came from our pipeline/team”, not just “it exists in our registry”
• Policy enforcement: gate deployments on verified signatures/attestations (Kubernetes admission, CI/CD checks)
• Auditability: prove what was deployed and when, and how it was built
• Better incident response: quickly answer “which builds include vulnerable component X?” when you have SBOMs attached to digests

Where do signatures/attestations live? OCI Artifacts + the Referrers API

Modern container signing doesn’t embed signatures “inside” an image. Instead, registries store additional OCI artifacts that refer to the image.

The OCI Distribution Spec defines a Referrers API (OCI v1.1) that lets a client ask:

“List all artifacts that refer to this subject digest.”

That makes signatures, SBOMs, provenance, scan reports, etc. first-class, discoverable artifacts associated with an image.

Azure Container Registry supports storing and discovering these supply-chain artifacts. Tools like ORAS can show the attached artifact graph, and ACR supports the OCI Referrers API in most scenarios.

Note: some environments may fall back to the older “referrers tag schema” approach instead of the Referrers API. For example, ACR’s documentation notes that some features (notably customer-managed key encrypted registries) may require fallback behavior.

Cosign vs Notation (quick comparison)

Both Cosign and Notation ultimately store “signatures as OCI artifacts” in registries, but they’re optimised for different workflows.

Cosign (Sigstore)

• Excellent developer experience for keyless signing using OIDC identities (Fulcio certificates) and transparency logging (Rekor)
• Strong support for attestations (provenance, SBOM predicates, etc.)
• Works well in modern CI/CD environments where identity is provided by an OIDC issuer (GitHub Actions, etc.)

Notation (Notary Project)

• Industry standardisation effort for OCI signing with an emphasis on X.509 / trust policies
• Strong fit for enterprise key management, including KMS/HSM-backed keys (for example via an Azure Key Vault plugin)
• Focused primarily on signing/verifying artifacts; attestations are commonly handled via separate OCI artifacts and tooling

If you’re building a modern “provenance + SBOM + policy” workflow, Cosign is often the fastest path. If you need a standardised signature format and strong enterprise policy/trust-store modeling (and/or a vendor-supported path), Notation is worth serious consideration.

Walkthrough: build, push, sign, and attach attestations in ACR

This walkthrough uses:

• Azure Container Registry (ACR)
• cosign for signing and attestations
• oras to discover and reason about the artifact graph (Referrers API)

Prerequisites

• Azure CLI (az) authenticated to your subscription
• ACR with permissions to push images and artifacts
• Tools:
  • cosign
  • oras
  • jq (optional, but helpful)
  • SBOM generator such as syft (optional)

On macOS you can typically install these with Homebrew (adjust as needed):

brew install cosign oras jq syft

1) Create an ACR and sign in

az login

export LOCATION=uksouth
export RG=rg-acr-signing-demo
export ACR_NAME=<your-unique-acr-name>

az group create -n "$RG" -l "$LOCATION"
az acr create -g "$RG" -n "$ACR_NAME" --sku Standard

export REGISTRY="$ACR_NAME.azurecr.io"
az acr login -n "$ACR_NAME"

2) Build and push an image (and capture the digest)

You can build locally with Docker and push, but for a clean “works anywhere” demo, ACR Tasks is handy.

export REPO=demo/net-monitor
export TAG=v1

# Example source repo used in Microsoft docs; swap this for your own Dockerfile repo.
export IMAGE_SOURCE=https://github.com/wabbit-networks/net-monitor.git#main

DIGEST=$(az acr build \
	-r "$ACR_NAME" \
	-t "$REGISTRY/${REPO}:$TAG" \
	"$IMAGE_SOURCE" \
	--no-logs \
	--query "outputImages[0].digest" -o tsv)

export IMAGE_DIGEST="$REGISTRY/${REPO}@$DIGEST"
echo "$IMAGE_DIGEST"

From here on, use $IMAGE_DIGEST (digest reference) for signing and attestations.

3) Sign the image with Cosign (keyless)

Keyless signing is Cosign’s default “batteries included” path.

cosign sign --yes "$IMAGE_DIGEST"

What to expect:

• A browser-based OIDC flow (unless you’re already in a CI environment with an OIDC token)
• A short-lived signing certificate issued by Sigstore’s Fulcio
• A transparency log entry in Rekor (by default)
• A signature stored in your registry as an OCI artifact referring to the image

4) Verify the Cosign signature

Verification should be part of your CI and/or cluster admission policy. The important idea is: verify not only “is it signed?” but “is it signed by the identity we trust?”.

For interactive keyless signing, the OIDC issuer is commonly:

OIDC_ISSUER="https://oauth2.sigstore.dev/auth"

And the identity is typically the OIDC subject (often your email). Example:

cosign verify \
	--certificate-identity "you@example.com" \
	--certificate-oidc-issuer "$OIDC_ISSUER" \
	"$IMAGE_DIGEST"

In CI (for example GitHub Actions), you’d instead verify against that workflow identity and issuer (for example https://token.actions.githubusercontent.com), often using the --certificate-identity-regexp flags to match your org/repo/workflow.

5) Generate an SBOM and attach it as an attestation (Cosign)

One straightforward pattern is:

1) Generate an SBOM file 2) Attest to it (sign a statement that includes it)

Generate an SBOM with Syft (SPDX JSON example):

syft packages "$IMAGE_DIGEST" -o spdx-json=sbom.spdx.json

Create an attestation for that SBOM:

cosign attest --yes \
	--predicate sbom.spdx.json \
	--type spdxjson \
	"$IMAGE_DIGEST"

Verify the attestation:

cosign verify-attestation \
	--certificate-identity "you@example.com" \
	--certificate-oidc-issuer "$OIDC_ISSUER" \
	--type spdxjson \
	"$IMAGE_DIGEST" \
	| jq -r '.[0].payload' \
	| base64 --decode

6) Discover what’s attached (OCI Referrers) with ORAS

This is the piece most people miss: once signatures/SBOMs/attestations are stored as OCI artifacts, you can query the graph.

oras discover -o tree "$IMAGE_DIGEST"

You should see the subject image at the top, with child artifact types beneath it (signatures, attestations, and any other artifacts you attach).

Alternative pattern: store the SBOM as an OCI artifact, then sign that

Sometimes you want the SBOM itself stored as a first-class artifact in the registry (so it can be pulled directly), and then you sign/attest to that artifact. OCI referrers support deep graphs: image → SBOM → signature.

Attach an SBOM file to the image as a referenced OCI artifact:

oras attach "$IMAGE_DIGEST" \
	--artifact-type sbom/example \
	./sbom.spdx.json:application/json

Discover again:

oras discover -o tree "$IMAGE_DIGEST"

This pattern is powerful when you want:

• direct pull of SBOMs (oras pull), separate from verification tooling
• to sign the SBOM artifact independently (for example with Notation)

Promoting images with their signatures/attestations

If you promote images across registries/environments, you should promote the artifact graph, not only the image manifest.

With ORAS you can copy an image and its referrers:

export TARGET_REPO="$REGISTRY/staging/$REPO"
oras copy -r "$IMAGE_DIGEST" "$TARGET_REPO:$TAG"

Then:

oras discover -o tree "$TARGET_REPO:$TAG"

Notation (brief practical example in Azure)

If you’re evaluating Notation in Azure, Microsoft provides a walkthrough that signs images in ACR using keys held in Azure Key Vault. The high-level flow looks like:

1) Install Notation + the Azure Key Vault plugin 2) Build/push image and sign by digest 3) Add certs to a trust store and create a trust policy 4) Verify

Signing example (Key Vault plugin; simplified):

notation sign \
	--signature-format cose \
	--id "$KEY_ID" \
	--plugin azure-kv \
	--plugin-config self_signed=true \
	"$IMAGE_DIGEST"

And list signatures:

notation ls "$IMAGE_DIGEST"

An important nuance called out in Microsoft’s guidance: Notation can store signatures using the referrers tag schema by default, and can also use the OCI Referrers API depending on configuration/registry capabilities.

Common pitfalls

• Always sign by digest (@sha256:...), not by tag.
• Verification must include identity constraints, not only “is signed?”.
• Plan where your trust roots live: developer keyless identity, CI workload identity, or enterprise PKI (Notation) are different trust models.
• Think about artifact lifecycle: deleting an image doesn’t always automatically delete every related artifact in every registry implementation.
• Referrers API vs tag schema: some environments may require fallback behavior; test your exact ACR configuration (including encryption features) if you’re standardising on Referrers API workflows.

Suggested “good default” pipeline shape

If you’re starting from scratch, a practical baseline is:

1) Build image 2) Push image 3) Generate SBOM 4) Create provenance (if your build system supports it) 5) Sign image digest 6) Attest SBOM/provenance to image digest 7) Enforce verification at deploy time (admission policy)

If you want to go further, add:

• environment-specific signatures (promotion gates)
• oras copy -r promotion of artifact graphs
• key management hardening (HSM/KMS), timestamping, and policy-as-code

What is Peep?

Peep is a system monitoring application for macOS that displays real-time information about your computer’s performance. Think of it as a modern, visually appealing alternative to Activity Monitor. It features:

• Real-time CPU and memory monitoring with animated gauges
• Network and disk I/O tracking with historical charts
• Process management with tree view and thread filtering
• Battery status with time remaining estimates
• Machine information including model name and macOS version

The tech stack combines Electron for the desktop shell, React and TypeScript for the UI, and Rust via Neon bindings for high-performance native system access.

Vibe Coding: A New Development Paradigm

“Vibe coding” is an emerging term for a development style where you describe what you want in natural language and iterate rapidly with AI assistance. Rather than meticulously planning every implementation detail upfront, you:

1. Describe the desired outcome - “Add battery information to the header”
2. Review the generated solution - AI provides working code
3. Refine through conversation - “The icon isn’t showing correctly for discharging state”
4. Iterate until satisfied - Continue the dialogue until it works perfectly

This approach flips traditional development on its head. Instead of spending hours reading documentation and debugging, you spend time describing and refining. The AI handles the boilerplate, the API lookups, and often catches edge cases you hadn’t considered.

The Development Journey

Starting Point: Architecture Decisions

The project began with a clear vision: a system monitor that felt native to macOS while being built with web technologies. The architecture emerged through conversation:

Me: “I want to build a system monitor for macOS using Electron”

Copilot: Suggested the full stack; Electron for the shell, React for the UI, and importantly, recommended Rust with Neon bindings for the native module rather than pure Node.js. This was crucial for performance when polling system metrics every few seconds.

The initial scaffolding; webpack config, TypeScript setup, IPC handlers, Rust/Neon integration, was generated in minutes. Tasks that would typically take half a day of configuration wrestling were done before my coffee got cold.

Copilot opens a pull request with a thorough explanation of the work that has been done. At this point I can pull the branch to my machine and do some QA.

The initial version created by Copilot looked like so:

If I find any bugs or want to make some minor improvements I can add a comment to the pull request.

glenthomas commented

@copilot the nodejs project is trying to import the Rust binary and is failing with error:

Failed to load native module: Error: Cannot find module ‘../../native/index.node’

Require stack:

/Users/glen.thomas/git/peep/dist/main.js

at Module._resolveFilename (node:internal/modules/cjs/loader:1390:15)

Copilot add the eyes emoji (👀) to to my comment to acknowledge my request and gets straight to work in a new session. A new comment is added to the pull request to let me know when the next stage of the work has been completed.

Copilot AI commented

Fixed in commit e1003c2. The issue was that the native module path was using a hardcoded relative require path that didn’t resolve correctly at runtime.

Changes made:

1. Updated src/main/main.ts to use path.join(__dirname, ‘..’, ‘native’, ‘index.node’) for proper path resolution
2. Added @types/node and @types/electron as dev dependencies (required for TypeScript compilation)
3. Updated build script to use npx cargo-cp-artifact to ensure the tool is available
4. Enhanced README troubleshooting section

The native module now loads correctly. From dist/main.js, it resolves to /native/index.node. Verified with test that shows the module loads successfully and can retrieve system info.

Feature Development: The Tree View Example

One of the more complex features was adding a process tree view that shows parent-child relationships between processes. Here’s how that conversation went:

Me: “Could we have an option to view process parent-child relationships?”

Within moments, I had:

• A buildProcessTree() function that constructs a hierarchy from the flat process list
• A renderProcessTree() function with proper indentation and expand/collapse
• Toggle buttons for switching between list and tree views
• State management for tracking expanded nodes

The implementation used the ppid (parent process ID) field that was already being collected by the Rust module. Copilot recognised the data was available and used it appropriately—something that would have required me to trace through the codebase manually.

Debugging: The Battery Icon Bug

Not everything worked perfectly on the first try, and that’s where the iterative nature of vibe coding shines. After adding battery status to the header, I noticed the icon wasn’t correct:

Me: “The battery state icon is not showing the correct icon. When the battery state is ‘discharging’, it shows the charging icon.”

Copilot identified the issue immediately: the getStateIcon() function was using includes() to match states, so “discharging” was matching the “charging” case first. The fix was simple—use exact matching with === and startsWith() instead.

This kind of bug could easily waste 20 minutes of console.log debugging. Instead, describing the symptom led directly to the fix.

The UI Polish Phase

As the core functionality stabilised, I shifted to UI refinements. This is where vibe coding particularly excels - making small adjustments is as easy as asking:

• “The columns, view type and show threads controls are stacked vertically, can they be stacked horizontally?” → Segmented button groups
• “On the CPU monitor and Memory monitor there is a bit of a vertical gap” → SVG viewport adjustments
• “Can we decrease the size of the gauges a little? Maybe 20%” → Proportional scaling of all gauge dimensions
• “The battery info is not properly centered in the header” → CSS Grid with 1fr auto 1fr

Each of these would typically involve researching CSS properties, testing various approaches, and iterating. With Copilot, I described the visual problem and received targeted solutions.

Native Integration Challenges

Some features required deeper system integration. When I asked about displaying the machine model name, the response included:

1. Using sysctl to read hw.model (e.g., “MacBookPro18,1”)
2. A mapping table translating hardware identifiers to friendly names
3. Integration with the existing get_os_info() Rust function

Similarly, adding the macOS marketing name (Sonoma, Sequoia, etc.) required mapping Darwin version numbers to release names—knowledge that would require research but was immediately available through Copilot.

When AI Knowledge Falls Short

It wasn’t all smooth sailing. One notable friction point came when working with the Rust sysinfo crate. Copilot confidently suggested methods and function calls that simply didn’t exist in the current version of the library.

For example, when implementing disk I/O monitoring per process, the AI suggested using methods that had either been renamed, deprecated, or never existed in the version I was using (0.37.x). The code would compile with errors like “method not found” or “no field named X on type Y”.

This is a fundamental limitation of AI coding assistants: their training data has a cutoff date, and libraries evolve. The sysinfo crate in particular has undergone significant API changes between versions, with methods being renamed and structs being reorganised.

The solution? I had to do things the old-fashioned way—pull up the sysinfo documentation on docs.rs, find the correct method names for my version, and either correct the AI’s suggestions or implement the fix myself. In some cases, I’d share the compiler error and the relevant documentation snippet, and Copilot would adjust. In others, it was faster to just fix it manually.

This experience reinforced an important lesson: AI assistants are collaborators, not oracles. They’re incredibly useful for getting 80% of the way there quickly, but you still need domain knowledge and the ability to debug when their information is stale or incorrect.

Self-Review: Getting Copilot to Improve Its Own Code

One powerful technique that addresses concerns about AI-generated code quality is using Copilot to review and improve the very code it has produced. Critics might argue that AI assistants lack sufficient context while working incrementally, potentially leading to suboptimal solutions. However, this misses a crucial capability: asking the AI to review the solution as a whole.

After Copilot had implemented the core functionality, I took a step back and asked:

Me: “Review the entire codebase and identify any issues, inefficiencies, or areas that could be improved.”

This prompted a comprehensive analysis that surfaced real problems I hadn’t noticed during rapid feature development:

🔴 High Priority Issues

1. Duplicate formatBytes function - Defined in 4 separate files (MemoryMonitor.tsx, DiskMonitor.tsx, NetworkMonitor.tsx, ProcessList.tsx), each with slightly different implementations
2. Type safety issues in App.tsx - Using any types for state variables like systemInfo, processes, and batteryInfo instead of proper interfaces
3. Preload types don’t match actual API - The ProcessInfo interface was missing many fields the Rust module actually returns (ppid, runTime, cpuTime, status, command, diskRead, diskWrite, isThread)

🟡 Medium Priority Issues

1. Native module type is any - No TypeScript declarations for the Rust bindings
2. Duplicate formatStorage function - Defined in both DiskMonitor.tsx and NetworkMonitor.tsx
3. ChartJS registered multiple times - Each monitor component was independently registering ChartJS components, should be done once at app level
4. Large inline styles in App.tsx header - The battery status section (188-275 lines) had extensive inline styles that could be extracted to CSS
5. ProcessList.tsx is 664 lines - Very large component that could be split into ProcessTable.tsx, ProcessControls.tsx, and ConfirmDialog.tsx

🟢 Low Priority

1. CSS variables defined but not fully used - Variables like --spacing-xs and --font-sm were defined but many inline styles used hardcoded values
2. Missing React.memo on monitor components - Could prevent unnecessary re-renders on gauge/chart components
3. Console.error for non-critical failures - Should use proper logging utility or remove in production

For each issue, Copilot provided context and suggested fixes. The conversation flowed naturally:

Me: “Fix the duplicate formatBytes functions”

Copilot: [Creates src/utils/formatters.ts with a single, comprehensive implementation and updates all 4 files to import it]

Me: “Add proper TypeScript types for the system info state”

Copilot: [Creates interface definitions matching the Rust module’s output and updates all any types to use them]

This self-review pattern is remarkably effective because:

1. Full context awareness - During review, the AI can see all the pieces together, not just the immediate task
2. Pattern recognition - It can spot anti-patterns across the codebase that emerge from incremental development
3. Best practices application - It can suggest idiomatic improvements that might have been missed during rapid feature development
4. Consistency checking - It identifies places where similar functionality is handled differently

The key is being explicit about the review scope. Rather than asking a vague “make it better,” specific prompts work best:

• “Review the React components for performance issues”
• “Check the Rust module for error handling problems”
• “Analyze the IPC communication for potential race conditions”
• “Review accessibility and ensure proper ARIA labels”

This approach transforms a potential weakness into a strength. Yes, AI might produce less-than-perfect code when working with limited context on individual features. But it excels at identifying those imperfections when given the full picture and asked to review holistically.

Setting Up the Release Pipeline

One area where AI assistance proved invaluable was configuring the build and release pipeline. CI/CD configuration is notoriously fiddly, with countless options and edge cases.

Me: “I would like to allow people finding this GitHub repository to be able to download and install the application. What options do I have?”

Copilot outlined several distribution strategies (GitHub Releases, Homebrew, Mac App Store) and recommended GitHub Releases with electron-builder for an open-source project. It then generated:

• A build.yml workflow for CI on every push
• A release.yml workflow triggered by version tags
• Electron-builder configuration for both Intel and Apple Silicon
• README updates with download instructions

When the workflows hit issues; wrong action names, permission errors, DMG creation failures on CI runners, each problem was solved through conversation:

• “The action dtolnay/rust-action@stable does not exist” → Switched to actions-rust-lang/setup-rust-toolchain@v1
• “Package ‘electron’ is only allowed in devDependencies” → Moved the dependency
• “hdiutil detach failing on CI” → Added retry logic for flaky DMG creation
• “403 Forbidden when creating release” → Added permissions: contents: write

Lessons Learned

When Vibe Coding Excels

1. Boilerplate and configuration - Webpack configs, GitHub Actions, package.json scripts
2. API integration - Knowing which Electron or React APIs to use
3. Cross-referencing - Using data from one part of the app in another
4. Bug diagnosis - Describing symptoms and getting targeted fixes
5. Refactoring - “Can you change this from flex to grid layout?”

When to Slow Down

1. Architecture decisions - Still worth thinking through yourself
2. Security-sensitive code - Review carefully, don’t blindly trust
3. Complex algorithms - Understand what’s generated, don’t just accept
4. Performance-critical paths - Verify the approach makes sense

The Collaboration Dynamic

The most effective pattern I found was treating Copilot as a highly knowledgeable pair programmer who happens to type very fast. I would:

1. Set context - Explain what exists and what I’m trying to achieve
2. Ask for options - “What are the ways to do X?”
3. Make decisions - Choose the approach that fits best
4. Request implementation - “Let’s go with option 2”
5. Iterate on details - “Can we also handle the edge case where…”

This maintains human agency over the architecture and design while leveraging AI for implementation speed.

The Results

Peep went from concept to released application in a remarkably short time. The final product includes:

• 5 monitoring components (CPU, Memory, Disk, Network, Processes)
• Native Rust module with 15+ exported functions
• Automated CI/CD with multi-architecture builds
• GitHub Releases with DMG and ZIP downloads
• Comprehensive documentation (README, CONTRIBUTING, CHANGELOG)

More importantly, the codebase is clean and maintainable. The AI-generated code follows consistent patterns, includes appropriate comments, and handles edge cases I might have missed.

Conclusion

Building Peep reinforced my belief that we’re at an inflection point in software development. The combination of vibe coding (describing intent in natural language) with tools like GitHub Copilot Chat doesn’t replace programming skill; it amplifies it.

You still need to understand what you’re building, make architectural decisions, and verify the results. But the mechanical translation of ideas into code? That’s increasingly handled by AI, freeing developers to focus on the creative and strategic aspects of software creation.

If you haven’t tried this approach, I encourage you to experiment. Start with a side project, describe what you want, and see where the conversation takes you. You might be surprised how quickly your ideas become reality.

Peep is open source and available on GitHub. Contributions welcome!

Docker Bake solves these challenges with a declarative approach to building container images. It’s a high-level build orchestration tool that lets you define complex build configurations in a single file, then execute them consistently across local development and CI/CD pipelines. Instead of maintaining dozens of docker build commands with slightly different flags, you define your entire build matrix once and let Bake handle the orchestration.

In this comprehensive guide, I’ll show you how to use Docker Bake to build production-ready container images. We’ll start with the fundamentals, progress through real-world examples of multi-platform and multi-tag builds, and finish with GitHub Actions workflows that demonstrate how to integrate Bake into your CI/CD pipeline. By the end, you’ll have the knowledge to replace your complex build scripts with maintainable, declarative configuration.

Understanding Docker Bake

Before diving into examples, it’s worth understanding what Docker Bake actually does and why it exists. Docker Bake is part of Docker Buildx, the extended build capabilities that replaced the classic Docker builder. Whilst docker buildx build handles building a single image, docker bake orchestrates building multiple related images in one operation.

Why Docker Bake Exists

Traditional Docker builds work well for simple use cases; a single Dockerfile producing a single image. But real world applications often need more complexity. You might need to build the same application for both AMD64 and ARM64 platforms. You might maintain separate images with different base operating systems. You might need to tag the same build with multiple version identifiers.

Without Bake, you’d write shell scripts that loop through platforms, base images, and tags, invoking docker build repeatedly with different arguments. These scripts become difficult to maintain, hard to test, and prone to subtle bugs where one combination of flags differs slightly from another.

Bake replaces these scripts with declarative configuration. You define what you want to build, not how to build it. The configuration format supports variables, inheritance, and composition, making it easier to maintain consistency across dozens of build targets.

Bake Configuration Format

Docker Bake supports three configuration formats: HCL (HashiCorp Configuration Language), JSON, and Docker Compose files. HCL is the most expressive and the format I recommend. It supports variables, functions, and a clean syntax that reads naturally.

A Bake file consists of targets, groups, and variables. Targets define individual build operations; which Dockerfile to use, what tags to apply, which platforms to build for. Groups collect multiple targets so you can build them together. Variables parameterise your configuration, allowing the same file to work across different environments.

How Bake Differs from Regular Builds

When you run docker buildx build, you’re building a single image. When you run docker buildx bake, you can build dozens of images in parallel, each with different configurations, all defined in your Bake file. Bake also provides better defaults for multi-platform builds, automatically configuring builders and handling the complexity of cross-compilation.

Another key difference is composability. Bake files can reference other Bake files, allowing you to split configuration across multiple files and compose them at build time. This makes it possible to maintain shared configuration whilst allowing project-specific overrides.

Basic Docker Bake Configuration

Let’s start with a simple example and build up complexity progressively. Create a file called docker-bake.hcl:

variable "TAG" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

group "default" {
  targets = ["app"]
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  tags       = ["${REGISTRY}/myapp:${TAG}"]
  platforms  = ["linux/amd64"]
}

This configuration defines a single target called app that builds your Dockerfile and tags it with a registry prefix and version tag. The default group contains this target, so running docker buildx bake without arguments builds it.

Variables make this configuration flexible. Override them at runtime without modifying the file:

docker buildx bake --set "*.TAG=1.0.0"

The * pattern applies the variable to all targets. You can also set variables through environment variables or by creating a docker-bake.override.hcl file that gets automatically merged with the main configuration.

Adding Context and Arguments

Real applications need build arguments for things like version numbers, build timestamps, or configuration values. Add them to your target:

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  tags       = ["${REGISTRY}/myapp:${TAG}"]
  platforms  = ["linux/amd64"]
  
  args = {
    VERSION    = "${TAG}"
    BUILD_DATE = timestamp()
    VCS_REF    = "git-sha"
  }
  
  labels = {
    "org.opencontainers.image.version"    = "${TAG}"
    "org.opencontainers.image.created"    = timestamp()
    "org.opencontainers.image.source"     = "https://github.com/myorg/myapp"
  }
}

The args block passes build arguments to your Dockerfile, accessible via ARG instructions. The labels block adds OCI-compliant metadata to your image. The timestamp() function generates the current timestamp, ensuring each build has accurate creation metadata.

Multi-Platform Builds

Multi-platform builds are where Docker Bake really shines. Building for multiple architectures with traditional Docker builds requires multiple commands, managing different builders, and complex scripting. Bake handles all of this automatically.

Configuring Platform Support

Update your target to build for both AMD64 and ARM64:

variable "TAG" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  tags       = ["${REGISTRY}/myapp:${TAG}"]
  platforms  = PLATFORMS
  
  args = {
    BUILDPLATFORM = ""
    TARGETPLATFORM = ""
  }
}

When you run docker buildx bake, Bake builds your image for both platforms in parallel, creating a multi-architecture manifest that automatically serves the correct image based on the host architecture.

The BUILDPLATFORM and TARGETPLATFORM arguments are automatically available in your Dockerfile when building multi-platform images. They let you customise build behaviour based on the target architecture:

FROM --platform=$BUILDPLATFORM golang:1.21 AS builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM

RUN echo "Building on $BUILDPLATFORM for $TARGETPLATFORM"

WORKDIR /app
COPY . .

# Use TARGETPLATFORM to set GOARCH appropriately
RUN case "$TARGETPLATFORM" in \
      "linux/amd64") GOARCH=amd64 ;; \
      "linux/arm64") GOARCH=arm64 ;; \
      "linux/arm/v7") GOARCH=arm GOARM=7 ;; \
    esac && \
    GOOS=linux go build -o myapp

FROM alpine:3.19
COPY --from=builder /app/myapp /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/myapp"]

Platform-Specific Optimisations

Sometimes you need platform-specific behaviour. For ARM builds, you might want to use a different base image or enable specific compiler optimisations. Create separate targets for each platform:

variable "TAG" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

target "app-amd64" {
  inherits   = ["app-common"]
  platforms  = ["linux/amd64"]
  tags       = ["${REGISTRY}/myapp:${TAG}-amd64"]
  
  args = {
    BASE_IMAGE = "alpine:3.19"
  }
}

target "app-arm64" {
  inherits   = ["app-common"]
  platforms  = ["linux/arm64"]
  tags       = ["${REGISTRY}/myapp:${TAG}-arm64"]
  
  args = {
    BASE_IMAGE = "arm64v8/alpine:3.19"
  }
}

target "app-common" {
  context    = "."
  dockerfile = "Dockerfile"
  
  args = {
    VERSION = "${TAG}"
  }
  
  labels = {
    "org.opencontainers.image.version" = "${TAG}"
  }
}

group "default" {
  targets = ["app-amd64", "app-arm64"]
}

This configuration uses inheritance through the inherits field. The app-common target defines shared configuration, whilst platform-specific targets override only what differs. Building the default group builds both platforms with their specific optimisations.

Multi-Tag Strategies

Production container workflows often need multiple tags pointing to the same image. You might tag a release as 1.2.3, 1.2, 1, and latest simultaneously. You might also tag builds with git commit SHAs for traceability. Bake makes this straightforward.

Semantic Versioning Tags

Here’s a configuration that tags images with multiple semantic version levels:

variable "VERSION" {
  default = "1.2.3"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "COMMIT_SHA" {
  default = "unknown"
}

function "semver_tags" {
  params = [version]
  result = [
    "${REGISTRY}/myapp:${version}",
    "${REGISTRY}/myapp:${regex("^[0-9]+\\.[0-9]+", version)}",
    "${REGISTRY}/myapp:${regex("^[0-9]+", version)}",
    "${REGISTRY}/myapp:latest"
  ]
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = semver_tags(VERSION)
  
  args = {
    VERSION    = VERSION
    COMMIT_SHA = COMMIT_SHA
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.revision" = COMMIT_SHA
  }
}

The semver_tags function takes a version like 1.2.3 and generates four tags: 1.2.3, 1.2, 1, and latest. This ensures users can pin to specific versions or track major/minor releases.

Environment-Specific Tags

Different environments often need different tagging strategies. Development builds might use branch names and commit SHAs, whilst production uses semantic versions:

variable "ENVIRONMENT" {
  default = "dev"
}

variable "VERSION" {
  default = "dev"
}

variable "BRANCH" {
  default = "main"
}

variable "COMMIT_SHA" {
  default = "unknown"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

function "dev_tags" {
  params = [branch, sha]
  result = [
    "${REGISTRY}/myapp:${branch}",
    "${REGISTRY}/myapp:${branch}-${sha}",
    "${REGISTRY}/myapp:dev"
  ]
}

function "prod_tags" {
  params = [version]
  result = [
    "${REGISTRY}/myapp:${version}",
    "${REGISTRY}/myapp:${regex("^[0-9]+\\.[0-9]+", version)}",
    "${REGISTRY}/myapp:${regex("^[0-9]+", version)}",
    "${REGISTRY}/myapp:latest"
  ]
}

target "app-dev" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64"]
  tags       = dev_tags(BRANCH, COMMIT_SHA)
}

target "app-prod" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = prod_tags(VERSION)
  
  output     = ["type=registry,push=true"]
}

group "dev" {
  targets = ["app-dev"]
}

group "prod" {
  targets = ["app-prod"]
}

Build development images with docker buildx bake dev and production images with docker buildx bake prod. Each group uses appropriate tags and platforms for its environment.

Multiple Base Images

Some applications need variants built from different base images. You might offer an Alpine-based image for minimal size and a Debian-based image for better compatibility. Or you might support multiple runtime versions—Node.js 18 and Node.js 20, for example.

Base Image Variants

Here’s a configuration that builds the same application with multiple base images:

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

target "app-alpine" {
  context    = "."
  dockerfile = "Dockerfile.alpine"
  platforms  = PLATFORMS
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-alpine",
    "${REGISTRY}/myapp:alpine"
  ]
  
  args = {
    BASE_IMAGE = "alpine:3.19"
    VERSION    = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.base.name" = "alpine:3.19"
  }
}

target "app-debian" {
  context    = "."
  dockerfile = "Dockerfile.debian"
  platforms  = PLATFORMS
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-debian",
    "${REGISTRY}/myapp:${VERSION}",
    "${REGISTRY}/myapp:latest"
  ]
  
  args = {
    BASE_IMAGE = "debian:bookworm-slim"
    VERSION    = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.base.name" = "debian:bookworm-slim"
  }
}

target "app-ubuntu" {
  context    = "."
  dockerfile = "Dockerfile.ubuntu"
  platforms  = PLATFORMS
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-ubuntu",
    "${REGISTRY}/myapp:ubuntu"
  ]
  
  args = {
    BASE_IMAGE = "ubuntu:22.04"
    VERSION    = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.base.name" = "ubuntu:22.04"
  }
}

group "default" {
  targets = ["app-alpine", "app-debian", "app-ubuntu"]
}

group "alpine" {
  targets = ["app-alpine"]
}

group "debian" {
  targets = ["app-debian"]
}

Running docker buildx bake builds all three variants. Running docker buildx bake alpine builds only the Alpine variant. Each variant has appropriate tags indicating its base image.

Single Dockerfile with Multiple Bases

If your Dockerfiles differ only in the base image, you can use a single Dockerfile with build arguments:

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

target "app-alpine" {
  inherits   = ["app-common"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-alpine"]
  
  args = {
    BASE_IMAGE = "alpine:3.19"
    IMAGE_VARIANT = "alpine"
  }
}

target "app-debian" {
  inherits   = ["app-common"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-debian", "${REGISTRY}/myapp:${VERSION}"]
  
  args = {
    BASE_IMAGE = "debian:bookworm-slim"
    IMAGE_VARIANT = "debian"
  }
}

target "app-common" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  
  args = {
    VERSION = VERSION
  }
}

group "default" {
  targets = ["app-alpine", "app-debian"]
}

Your Dockerfile uses the BASE_IMAGE argument:

ARG BASE_IMAGE=alpine:3.19
FROM ${BASE_IMAGE}

ARG VERSION
ARG IMAGE_VARIANT

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.variant="${IMAGE_VARIANT}"

WORKDIR /app
COPY . .

RUN if [ "${IMAGE_VARIANT}" = "alpine" ]; then \
      apk add --no-cache ca-certificates; \
    else \
      apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*; \
    fi

ENTRYPOINT ["/app/myapp"]

This approach reduces duplication when the only difference between variants is the base image and package manager.

Runtime Version Matrix

Applications often need to support multiple runtime versions. A Python application might support Python 3.10, 3.11, and 3.12. A Node.js application might support Node 18, 20, and 21. Bake makes it easy to build a complete matrix.

Python Version Matrix Example

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

variable "PYTHON_VERSIONS" {
  default = ["3.10", "3.11", "3.12"]
}

target "app-python-310" {
  inherits   = ["app-python-common"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python3.10",
    "${REGISTRY}/myapp:python3.10"
  ]
  
  args = {
    PYTHON_VERSION = "3.10"
    PYTHON_IMAGE = "python:3.10-slim"
  }
}

target "app-python-311" {
  inherits   = ["app-python-common"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python3.11",
    "${REGISTRY}/myapp:python3.11"
  ]
  
  args = {
    PYTHON_VERSION = "3.11"
    PYTHON_IMAGE = "python:3.11-slim"
  }
}

target "app-python-312" {
  inherits   = ["app-python-common"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python3.12",
    "${REGISTRY}/myapp:${VERSION}",
    "${REGISTRY}/myapp:latest"
  ]
  
  args = {
    PYTHON_VERSION = "3.12"
    PYTHON_IMAGE = "python:3.12-slim"
  }
}

target "app-python-common" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = PLATFORMS
  
  args = {
    VERSION = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
  }
}

group "default" {
  targets = ["app-python-310", "app-python-311", "app-python-312"]
}

group "python-latest" {
  targets = ["app-python-312"]
}

The corresponding Dockerfile:

ARG PYTHON_IMAGE=python:3.12-slim
FROM ${PYTHON_IMAGE}

ARG VERSION
ARG PYTHON_VERSION

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.python.version="${PYTHON_VERSION}"

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Advanced Bake Patterns

Once you’re comfortable with basic Bake configurations, several advanced patterns can make your builds even more powerful.

Matrix Builds with Dynamic Targets

For large build matrices, manually defining every combination becomes tedious. Whilst Bake doesn’t support true matrix generation in HCL, you can use environment variables and multiple Bake files to achieve similar results.

Create a base configuration docker-bake.hcl:

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PYTHON_VERSION" {
  default = "3.12"
}

variable "BASE_VARIANT" {
  default = "slim"
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python${PYTHON_VERSION}-${BASE_VARIANT}"
  ]
  
  args = {
    PYTHON_IMAGE = "python:${PYTHON_VERSION}-${BASE_VARIANT}"
    VERSION      = VERSION
  }
}

Then use a shell script to invoke Bake multiple times with different variables:

#!/bin/bash
set -e

VERSIONS=("3.10" "3.11" "3.12")
VARIANTS=("slim" "alpine")

for version in "${VERSIONS[@]}"; do
  for variant in "${VARIANTS[@]}"; do
    echo "Building Python ${version} ${variant}"
    docker buildx bake \
      --set "*.PYTHON_VERSION=${version}" \
      --set "*.BASE_VARIANT=${variant}" \
      app
  done
done

Conditional Building

Sometimes you want to build certain targets only in specific conditions. Whilst Bake doesn’t support conditionals directly, you can use groups and selective building:

variable "BUILD_ARCH" {
  default = "all"
}

target "app-amd64" {
  inherits   = ["app-common"]
  platforms  = ["linux/amd64"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-amd64"]
}

target "app-arm64" {
  inherits   = ["app-common"]
  platforms  = ["linux/arm64"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-arm64"]
}

target "app-common" {
  context    = "."
  dockerfile = "Dockerfile"
}

group "all" {
  targets = ["app-amd64", "app-arm64"]
}

group "amd64" {
  targets = ["app-amd64"]
}

group "arm64" {
  targets = ["app-arm64"]
}

Build only AMD64 with docker buildx bake amd64, only ARM64 with docker buildx bake arm64, or both with docker buildx bake all.

Build Secrets and SSH Forwarding

Production builds often need access to private dependencies or repositories. Bake supports BuildKit secrets and SSH forwarding:

variable "VERSION" {
  default = "latest"
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = ["myapp:${VERSION}"]
  
  secret = [
    "id=npm_token,env=NPM_TOKEN",
    "id=github_token,env=GITHUB_TOKEN"
  ]
  
  ssh = ["default"]
}

In your Dockerfile, mount secrets:

FROM node:20-alpine

WORKDIR /app

COPY package*.json ./

# Mount npm token as secret for private registry access
RUN --mount=type=secret,id=npm_token \
    echo "//registry.npmjs.org/:_authToken=$(cat /run/secrets/npm_token)" > .npmrc && \
    npm ci && \
    rm .npmrc

COPY . .

RUN npm run build

CMD ["npm", "start"]

For SSH access to private Git repositories:

FROM node:20-alpine

RUN apk add --no-cache git openssh-client

WORKDIR /app

# Mount SSH socket for private repo access
RUN --mount=type=ssh \
    mkdir -p ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts && \
    npm install git+ssh://git@github.com/myorg/private-package.git

COPY . .

CMD ["npm", "start"]

GitHub Actions Integration

Docker Bake integrates seamlessly with GitHub Actions, providing a clean way to build and push container images in CI/CD pipelines.

Basic GitHub Actions Workflow

Create .github/workflows/docker-build.yml:

name: Build and Push Docker Images

on:
  push:
    branches:
      - main
      - develop
    tags:
      - 'v*'
  pull_request:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract metadata
        id: meta
        run: |
          VERSION=latest
          if [[ $GITHUB_REF == refs/tags/v* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          elif [[ $GITHUB_REF == refs/heads/main ]]; then
            VERSION=main
          elif [[ $GITHUB_REF == refs/heads/* ]]; then
            VERSION=${GITHUB_REF#refs/heads/}
          fi
          
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
          echo "commit_sha=${GITHUB_SHA::8}" >> $GITHUB_OUTPUT
          echo "build_date=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
      
      - name: Build images
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          set: |
            *.VERSION=$
            *.COMMIT_SHA=$
            *.REGISTRY=$/$
          push: $

This workflow builds your images on every push and pull request, pushing only when changes are merged to main or when tags are created.

Multi-Platform Builds in GitHub Actions

For multi-platform builds, configure QEMU emulation:

name: Build Multi-Platform Images

on:
  push:
    tags:
      - 'v*'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          platforms: 'arm64,arm'
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract version
        id: version
        run: |
          VERSION=${GITHUB_REF#refs/tags/v}
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
      
      - name: Build and push multi-platform images
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          targets: default
          set: |
            *.VERSION=$
            *.REGISTRY=$/$
            *.PLATFORMS=linux/amd64,linux/arm64
          push: true

Matrix Builds in GitHub Actions

For building multiple variants, use GitHub Actions matrix strategy:

name: Build Image Matrix

on:
  push:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    
    strategy:
      matrix:
        python-version: ['3.10', '3.11', '3.12']
        base-variant: ['slim', 'alpine']
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Build and push
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          targets: app
          set: |
            *.PYTHON_VERSION=$
            *.BASE_VARIANT=$
            *.REGISTRY=$/$
          push: true

This workflow builds all combinations of Python versions and base variants in parallel.

Build Caching in CI

BuildKit caching dramatically speeds up CI builds. Configure cache exports and imports:

name: Build with Caching

on:
  push:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Build and push with cache
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          set: |
            *.REGISTRY=$/$
            *.cache-from=type=registry,ref=$/$:buildcache
            *.cache-to=type=registry,ref=$/$:buildcache,mode=max
          push: true

Update your docker-bake.hcl to support cache configuration:

variable "CACHE_FROM" {
  default = ""
}

variable "CACHE_TO" {
  default = ""
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  
  cache-from = CACHE_FROM != "" ? [CACHE_FROM] : []
  cache-to   = CACHE_TO != "" ? [CACHE_TO] : []
}

Reusable Workflow for Multiple Repositories

Create a reusable workflow that can be used across multiple repositories. In your organisation’s .github repository, create .github/workflows/docker-bake-reusable.yml:

name: Reusable Docker Bake Workflow

on:
  workflow_call:
    inputs:
      bake-file:
        description: 'Path to docker-bake.hcl file'
        required: false
        type: string
        default: './docker-bake.hcl'
      bake-target:
        description: 'Bake target to build'
        required: false
        type: string
        default: 'default'
      registry:
        description: 'Container registry'
        required: false
        type: string
        default: 'ghcr.io'
      platforms:
        description: 'Platforms to build for'
        required: false
        type: string
        default: 'linux/amd64,linux/arm64'
      push:
        description: 'Push images to registry'
        required: false
        type: boolean
        default: true
    secrets:
      registry-username:
        description: 'Registry username'
        required: false
      registry-password:
        description: 'Registry password'
        required: true

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract metadata
        id: meta
        run: |
          VERSION=latest
          if [[ $GITHUB_REF == refs/tags/v* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          elif [[ $GITHUB_REF == refs/heads/main ]]; then
            VERSION=main
          fi
          
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
          echo "commit_sha=${GITHUB_SHA::8}" >> $GITHUB_OUTPUT
      
      - name: Build and push
        uses: docker/bake-action@v4
        with:
          files: $
          targets: $
          set: |
            *.VERSION=$
            *.COMMIT_SHA=$
            *.REGISTRY=$/$
            *.PLATFORMS=$
          push: $

Use it in your application repositories:

name: Build Docker Images

on:
  push:
    branches:
      - main
    tags:
      - 'v*'

jobs:
  build:
    uses: myorg/.github/.github/workflows/docker-bake-reusable.yml@main
    with:
      platforms: 'linux/amd64,linux/arm64'
      push: true
    secrets:
      registry-password: $

Complete Real-World Example

Let’s put everything together with a complete example: a Node.js application that supports multiple Node versions, multiple platforms, and builds both development and production variants.

Project Structure

.
├── docker-bake.hcl
├── docker-bake.override.hcl.example
├── Dockerfile
├── package.json
├── src/
│   └── app.js
└── .github/
    └── workflows/
        └── docker.yml

docker-bake.hcl

variable "VERSION" {
  default = "dev"
}

variable "REGISTRY" {
  default = "ghcr.io/myorg"
}

variable "COMMIT_SHA" {
  default = "unknown"
}

variable "BUILD_DATE" {
  default = ""
}

variable "NODE_VERSIONS" {
  default = ["18", "20", "21"]
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

function "semver_tags" {
  params = [registry, app, version, node_version]
  result = [
    "${registry}/${app}:${version}-node${node_version}",
    "${registry}/${app}:node${node_version}"
  ]
}

function "prod_tags" {
  params = [registry, app, version, node_version]
  result = concat(
    semver_tags(registry, app, version, node_version),
    node_version == "20" ? [
      "${registry}/${app}:${version}",
      "${registry}/${app}:latest"
    ] : []
  )
}

# Node 18 targets
target "app-node18-dev" {
  inherits   = ["node-common"]
  tags       = ["${REGISTRY}/myapp:dev-node18"]
  args = {
    NODE_VERSION = "18"
    NODE_ENV = "development"
  }
  platforms = ["linux/amd64"]
}

target "app-node18-prod" {
  inherits   = ["node-common"]
  tags       = prod_tags(REGISTRY, "myapp", VERSION, "18")
  args = {
    NODE_VERSION = "18"
    NODE_ENV = "production"
  }
  output = ["type=registry,push=true"]
}

# Node 20 targets
target "app-node20-dev" {
  inherits   = ["node-common"]
  tags       = ["${REGISTRY}/myapp:dev-node20", "${REGISTRY}/myapp:dev"]
  args = {
    NODE_VERSION = "20"
    NODE_ENV = "development"
  }
  platforms = ["linux/amd64"]
}

target "app-node20-prod" {
  inherits   = ["node-common"]
  tags       = prod_tags(REGISTRY, "myapp", VERSION, "20")
  args = {
    NODE_VERSION = "20"
    NODE_ENV = "production"
  }
  output = ["type=registry,push=true"]
}

# Node 21 targets
target "app-node21-dev" {
  inherits   = ["node-common"]
  tags       = ["${REGISTRY}/myapp:dev-node21"]
  args = {
    NODE_VERSION = "21"
    NODE_ENV = "development"
  }
  platforms = ["linux/amd64"]
}

target "app-node21-prod" {
  inherits   = ["node-common"]
  tags       = prod_tags(REGISTRY, "myapp", VERSION, "21")
  args = {
    NODE_VERSION = "21"
    NODE_ENV = "production"
  }
  output = ["type=registry,push=true"]
}

# Common target configuration
target "node-common" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = PLATFORMS
  
  args = {
    VERSION    = VERSION
    COMMIT_SHA = COMMIT_SHA
    BUILD_DATE = BUILD_DATE != "" ? BUILD_DATE : timestamp()
  }
  
  labels = {
    "org.opencontainers.image.version"    = VERSION
    "org.opencontainers.image.revision"   = COMMIT_SHA
    "org.opencontainers.image.created"    = BUILD_DATE != "" ? BUILD_DATE : timestamp()
    "org.opencontainers.image.source"     = "https://github.com/myorg/myapp"
    "org.opencontainers.image.vendor"     = "MyOrg"
  }
  
  cache-from = ["type=registry,ref=${REGISTRY}/myapp:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/myapp:buildcache,mode=max"]
}

# Groups for different scenarios
group "dev" {
  targets = ["app-node18-dev", "app-node20-dev", "app-node21-dev"]
}

group "prod" {
  targets = ["app-node18-prod", "app-node20-prod", "app-node21-prod"]
}

group "node20" {
  targets = ["app-node20-dev", "app-node20-prod"]
}

group "default" {
  targets = ["app-node20-dev"]
}

Dockerfile

ARG NODE_VERSION=20
FROM node:${NODE_VERSION}-alpine AS base

ARG VERSION
ARG COMMIT_SHA
ARG BUILD_DATE
ARG NODE_ENV=production

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.revision="${COMMIT_SHA}"
LABEL org.opencontainers.image.created="${BUILD_DATE}"

WORKDIR /app

# Dependencies stage
FROM base AS dependencies

COPY package*.json ./

RUN if [ "$NODE_ENV" = "production" ]; then \
      npm ci --only=production; \
    else \
      npm ci; \
    fi

# Build stage (for TypeScript or other build steps)
FROM dependencies AS build

COPY . .

RUN if [ "$NODE_ENV" = "production" ]; then \
      npm run build 2>/dev/null || true; \
    fi

# Production stage
FROM base AS production

ENV NODE_ENV=production

# Copy only production dependencies
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist 2>/dev/null || true
COPY package*.json ./
COPY src ./src

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001 && \
    chown -R nodejs:nodejs /app

USER nodejs

EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})"

CMD ["node", "src/app.js"]

# Development stage
FROM dependencies AS development

ENV NODE_ENV=development

COPY . .

USER node

EXPOSE 3000

CMD ["npm", "run", "dev"]

GitHub Actions Workflow

name: Build and Push Docker Images

on:
  push:
    branches:
      - main
      - develop
    tags:
      - 'v*'
  pull_request:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build-dev:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request' || github.ref == 'refs/heads/develop'
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Build development images
        uses: docker/bake-action@v4
        with:
          files: ./docker-bake.hcl
          targets: dev
          set: |
            *.REGISTRY=$/$
            *.COMMIT_SHA=$
          push: $
  
  build-prod:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          platforms: 'arm64'
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract version
        id: version
        run: |
          if [[ $GITHUB_REF == refs/tags/v* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          else
            VERSION=main-${GITHUB_SHA::8}
          fi
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
      
      - name: Build and push production images
        uses: docker/bake-action@v4
        with:
          files: ./docker-bake.hcl
          targets: prod
          set: |
            *.VERSION=$
            *.COMMIT_SHA=$
            *.REGISTRY=$/$
            *.BUILD_DATE=$
          push: true

Best Practices and Tips

After working with Docker Bake across numerous projects, several patterns have emerged as particularly effective.

Keep Bake Files Declarative

Resist the temptation to add too much logic to your Bake files. They work best when they declare what you want to build, not how to build it. Complex conditionals and computations belong in CI scripts or separate tooling, not in Bake configuration.

Use Inheritance Extensively

The inherits field is one of Bake’s most powerful features. Define common configuration once in a base target, then inherit from it in specific targets. This reduces duplication and ensures consistency across related builds.

Organise Targets into Logical Groups

Groups make it easy to build related targets together. Create groups for different scenarios: dev for development builds, prod for production, ci for continuous integration. Users can then run docker buildx bake dev without needing to know which specific targets that includes.

Version Your Bake Files

Treat your docker-bake.hcl files as code. Version them in Git, review changes through pull requests, and test modifications before merging. Bake configuration changes can have significant impact on your build process.

Document Your Targets

Add comments explaining what each target does and when to use it. Future maintainers (including yourself) will appreciate understanding the purpose of app-python-311-alpine-optimised without having to reverse-engineer the configuration.

Use Override Files for Local Development

Create a docker-bake.override.hcl.example file that developers can copy to docker-bake.override.hcl for local customisation. Bake automatically merges override files, allowing developers to adjust registry paths or platforms without modifying the main configuration.

# docker-bake.override.hcl.example
variable "REGISTRY" {
  default = "localhost:5000"
}

variable "PLATFORMS" {
  default = ["linux/amd64"]  # Build only local platform for speed
}

Validate Your Bake Files

Before committing changes, validate your Bake files:

docker buildx bake --print

This shows the resolved configuration without actually building anything, catching syntax errors and misconfigurations early.

Cache Aggressively in CI

Build caching can reduce CI build times from minutes to seconds. Configure both cache-from and cache-to in your targets and ensure your CI environment pushes cache layers to a registry.

Monitor Build Times

Track how long your builds take and optimise the slow ones. Multi-stage builds with effective layer caching make the biggest difference. BuildKit’s build statistics show where time is spent.

Troubleshooting Common Issues

Even with careful configuration, you’ll occasionally encounter issues. Here are solutions to the most common problems.

Bake Can’t Find Dockerfile

Error: failed to solve: failed to read dockerfile

This usually means the context or dockerfile path is wrong. Remember that paths in Bake files are relative to the Bake file’s location, not your current directory. Use ./ explicitly:

target "app" {
  context    = "."
  dockerfile = "./Dockerfile"
}

Platforms Not Building

If multi-platform builds fail, ensure you have the correct builder:

docker buildx create --name multiplatform --driver docker-container --use
docker buildx inspect --bootstrap

Then verify your builder supports the platforms you need:

docker buildx inspect multiplatform

Cache Not Working

If builds aren’t using cache despite configuration, check that:

1. Cache references are accessible (correct registry, authentication)
2. You’re using mode=max for cache-to to cache all layers
3. Your Dockerfile uses layer caching effectively (copy dependencies before code)

Variables Not Resolving

If variables show as literal strings instead of values, check that:

1. You’re using ${} syntax, not $ alone: ${VERSION}, not $VERSION
2. Variables are defined before use
3. You’re not mixing environment variables with Bake variables (prefix env vars with $)

Build Fails in CI But Works Locally

This often indicates environmental differences:

1. Check platform architecture (CI might be different from local)
2. Verify secrets and SSH keys are available in CI
3. Ensure network access for downloading dependencies
4. Check resource limits (CI runners might have less memory/CPU)

Conclusion

Docker Bake transforms container image builds from imperative scripts into declarative configuration. Instead of maintaining complex shell scripts that orchestrate dozens of docker build commands, you define your complete build matrix in a single file that serves as documentation, automation, and contract all at once.

The patterns we’ve explored—multi-platform builds, multiple base images, runtime version matrices, sophisticated tagging strategies—all become manageable through Bake’s inheritance, functions, and groups. Your build configuration becomes more maintainable, easier to understand, and more reliable.

For production container workflows, Bake isn’t just a convenience, it’s a force multiplier that lets you maintain complex build matrices without drowning in complexity. The GitHub Actions integrations show how Bake fits naturally into modern CI/CD pipelines, providing consistent builds from local development through production deployment.

As your containerisation needs grow, Docker Bake grows with you. Start simple with basic targets and groups, then progressively add platforms, variants, and optimisations as requirements evolve. The declarative configuration ensures that complexity remains manageable no matter how large your build matrix becomes.

The investment in learning Docker Bake pays dividends quickly. Your builds become faster through better caching, more reliable through consistent configuration, and more maintainable through clear declarative structure. You’ve built a foundation that will scale from a handful of images to hundreds of build variants without requiring fundamental rewrites.

The reality is that containers share the host kernel with every other container on the same machine. A vulnerability in that kernel affects every container. A container escape gives an attacker access to the entire host and potentially every other workload running there. Understanding these risks—and the mechanisms Linux provides to mitigate them—is essential for anyone running containers in production.

In this comprehensive guide, I’ll walk you through the fundamental security concepts that underpin container isolation, the practical measures you can implement to harden your containerised applications, and the tools and workflows that make security a natural part of your development process rather than an afterthought bolted on at deployment time.

Understanding Container Isolation

Before we can secure containers effectively, we need to understand how they achieve isolation in the first place. Containers aren’t virtual machines—they don’t run separate kernels or emulate hardware. Instead, they use Linux kernel features to create isolated environments that share the same kernel whilst appearing to be separate systems.

The Shared Kernel Model

Every container on a host shares the same Linux kernel. When you run docker run ubuntu:22.04, you’re not running an Ubuntu kernel—you’re running the host’s kernel with an Ubuntu userspace. This is fundamentally different from virtual machines, where each VM runs its own kernel instance.

This shared kernel model brings both efficiency and risk. Containers start in milliseconds because there’s no kernel to boot. They use less memory because there’s no kernel overhead per container. But a kernel vulnerability affects every container on the host, and a container escape potentially compromises every workload.

Understanding this model shapes how we think about container security. We’re not securing isolated systems; we’re securing processes that share resources with other processes whilst maintaining the illusion of isolation.

Linux Namespaces

Namespaces are the primary isolation mechanism for containers. They partition kernel resources so that one set of processes sees one set of resources whilst another set of processes sees a different set. Linux provides several namespace types, each isolating a different aspect of the system.

The PID namespace isolates process IDs. Inside a container, processes see themselves as PID 1, 2, 3, and so on, but from the host’s perspective, these same processes have entirely different PIDs. This prevents containers from seeing or signalling processes in other containers.

The network namespace isolates network interfaces, routing tables, and firewall rules. Each container gets its own network stack, appearing to have dedicated network interfaces even though the physical hardware is shared.

The mount namespace isolates filesystem mount points. A container sees its own root filesystem without access to the host’s filesystem or other containers’ filesystems.

The UTS namespace isolates hostname and domain name. Each container can have its own hostname without affecting others.

The IPC namespace isolates inter-process communication resources like shared memory segments and message queues. This prevents containers from communicating through these mechanisms unless explicitly allowed.

The user namespace maps user and group IDs inside the container to different IDs outside. Root inside the container can be mapped to an unprivileged user on the host, significantly reducing the impact of container escapes.

Here’s how you can inspect namespaces for a running container:

# Get the container's PID on the host
CONTAINER_PID=$(docker inspect --format '' my-container)

# List the namespaces for this process
ls -la /proc/$CONTAINER_PID/ns/

# Compare with host namespaces
ls -la /proc/1/ns/

Control Groups (cgroups)

While namespaces provide isolation of what a container can see, cgroups control what resources a container can use. They limit and account for CPU, memory, disk I/O, and network bandwidth consumption.

Without cgroups, a single container could consume all available memory, causing the kernel’s out-of-memory killer to terminate other containers or even critical host processes. A container could monopolise CPU, starving other workloads. Cgroups prevent this resource starvation.

# View cgroup limits for a container
docker inspect --format '' my-container
docker inspect --format '' my-container

# Set resource limits when running a container
docker run -d \
  --memory="512m" \
  --memory-swap="512m" \
  --cpus="1.5" \
  --pids-limit=100 \
  myapp:latest

The --memory flag limits total memory. The --memory-swap flag set equal to --memory disables swap, preventing the container from using swap space when memory is exhausted. The --cpus flag limits CPU usage to 1.5 cores. The --pids-limit flag prevents fork bombs by limiting the number of processes.

Linux Capabilities

Traditional Unix security divides the world into two categories: root, which can do anything, and everyone else, which is heavily restricted. This is too coarse-grained for containers. You might need a container that can bind to privileged ports but can’t load kernel modules.

Linux capabilities split root’s powers into distinct units that can be granted independently. Instead of running as root with all powers, you can run as root with only the specific capabilities needed.

# Run a container with minimal capabilities
docker run -d \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  myapp:latest

# List capabilities of a running container
docker inspect --format '' my-container
docker inspect --format '' my-container

Key capabilities to understand:

• CAP_NET_BIND_SERVICE: Bind to ports below 1024
• CAP_NET_RAW: Use raw sockets (needed for ping)
• CAP_SYS_ADMIN: A catch-all for various admin operations (avoid granting this)
• CAP_SYS_PTRACE: Trace processes (needed for debugging)
• CAP_CHOWN: Change file ownership
• CAP_SETUID / CAP_SETGID: Change user/group IDs

The principle of least privilege demands dropping all capabilities and adding back only those specifically required. Most applications need far fewer capabilities than Docker grants by default.

Seccomp Profiles

Seccomp (secure computing mode) filters system calls that a process can make. The Linux kernel provides hundreds of syscalls, but most applications need only a fraction of them. Seccomp profiles whitelist or blacklist specific syscalls, reducing the kernel attack surface available to a container.

Docker applies a default seccomp profile that blocks around 44 syscalls considered dangerous, including reboot, mount, and kexec_load. You can create custom profiles for tighter restrictions:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "defaultErrnoRet": 1,
  "architectures": [
    "SCMP_ARCH_X86_64",
    "SCMP_ARCH_AARCH64"
  ],
  "syscalls": [
    {
      "names": [
        "accept",
        "accept4",
        "bind",
        "clone",
        "close",
        "connect",
        "dup",
        "dup2",
        "execve",
        "exit",
        "exit_group",
        "fcntl",
        "fstat",
        "futex",
        "getpid",
        "getsockname",
        "getsockopt",
        "listen",
        "mmap",
        "mprotect",
        "munmap",
        "open",
        "openat",
        "read",
        "recvfrom",
        "rt_sigaction",
        "rt_sigprocmask",
        "sendto",
        "setsockopt",
        "socket",
        "write"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Apply a custom seccomp profile:

docker run -d \
  --security-opt seccomp=/path/to/profile.json \
  myapp:latest

To generate a seccomp profile for your application, you can use tools like strace to observe which syscalls your application actually makes, then create a whitelist profile.

Image Security

Container images are the foundation of your containerised applications. A vulnerable or malicious base image compromises everything built upon it. Securing your images starts with choosing the right base and extends through scanning, signing, and supply chain verification.

Choosing Secure Base Images

The base image you choose dramatically affects your security posture. A full Ubuntu image contains thousands of packages, each potentially harbouring vulnerabilities. A minimal image contains only what’s necessary, reducing attack surface significantly.

Distroless Images

Google’s distroless images contain only your application and its runtime dependencies—no shell, no package manager, no unnecessary utilities. An attacker who gains code execution can’t spawn a shell because there isn’t one.

# Build stage
FROM golang:1.21 AS builder

WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o myapp

# Runtime stage - distroless
FROM gcr.io/distroless/static-debian12

COPY --from=builder /app/myapp /
USER nonroot:nonroot

ENTRYPOINT ["/myapp"]

Alpine Linux

Alpine uses musl libc instead of glibc and provides a minimal base around 5MB. It includes a package manager but far fewer default packages than traditional distributions.

FROM alpine:3.19

RUN apk add --no-cache ca-certificates tzdata

COPY myapp /usr/local/bin/
USER nobody:nobody

ENTRYPOINT ["/usr/local/bin/myapp"]

Scratch Images

The scratch image is literally empty; no files, no filesystem, nothing. It’s suitable for statically compiled binaries that include everything they need.

FROM golang:1.21 AS builder

WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o myapp

FROM scratch

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app/myapp /

ENTRYPOINT ["/myapp"]

Vulnerability Scanning

Vulnerability scanners analyse your images against databases of known vulnerabilities (CVEs). They identify vulnerable packages and provide severity ratings to help prioritise remediation.

Trivy

Trivy is a comprehensive, fast scanner that checks for OS package vulnerabilities, language-specific dependencies, misconfigurations, and secrets.

# Scan an image
trivy image myapp:latest

# Scan with specific severity threshold
trivy image --severity HIGH,CRITICAL myapp:latest

# Scan and fail if vulnerabilities found (useful for CI)
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a Dockerfile for misconfigurations
trivy config Dockerfile

# Generate SBOM while scanning
trivy image --format spdx-json --output sbom.spdx.json myapp:latest

Grype

Grype from Anchore focuses specifically on vulnerability scanning with excellent accuracy and speed.

# Scan an image
grype myapp:latest

# Output as JSON for processing
grype -o json myapp:latest > vulnerabilities.json

# Fail on specific severity
grype myapp:latest --fail-on critical

Snyk

Snyk provides vulnerability scanning with remediation advice and integrates well with development workflows.

# Scan an image
snyk container test myapp:latest

# Monitor for new vulnerabilities
snyk container monitor myapp:latest

# Scan and provide fix recommendations
snyk container test myapp:latest --file=Dockerfile

Multi-Stage Builds for Security

Multi-stage builds are a security feature as much as an optimisation. They ensure build tools, source code, test data, and secrets used during build don’t end up in your final image.

# Stage 1: Dependencies
FROM node:20-alpine AS dependencies

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Stage 2: Build
FROM node:20-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build
RUN npm run test

# Stage 3: Production - only runtime artifacts
FROM node:20-alpine AS production

RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

# Copy only production dependencies
COPY --from=dependencies --chown=nodejs:nodejs /app/node_modules ./node_modules

# Copy only built artifacts
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/package.json ./

USER nodejs

EXPOSE 3000

CMD ["node", "dist/server.js"]

Notice what doesn’t appear in the final image: npm (we use node directly), the full node_modules with dev dependencies, source TypeScript files, test files, and any secrets used during the build process. The final image contains only what’s needed to run the application.

Image Signing and Verification

Image signing cryptographically proves that an image came from a trusted source and hasn’t been tampered with. Sigstore’s Cosign has become the de facto standard for container image signing.

# Generate a key pair (for traditional key-based signing)
cosign generate-key-pair

# Sign an image with a key
cosign sign --key cosign.key myregistry/myapp:v1.0.0

# Sign using keyless signing (recommended)
# This uses OIDC identity from your CI provider
cosign sign myregistry/myapp:v1.0.0

# Verify a signature
cosign verify --key cosign.pub myregistry/myapp:v1.0.0

# Verify keyless signature
cosign verify \
  --certificate-identity=https://github.com/myorg/myapp/.github/workflows/build.yml@refs/tags/v1.0.0 \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  myregistry/myapp:v1.0.0

Keyless signing with Sigstore is particularly powerful for CI/CD. It uses your CI provider’s OIDC identity to sign images, eliminating the need to manage signing keys. The signature includes claims about who built the image and in what context, providing strong provenance guarantees.

Software Bill of Materials (SBOM)

An SBOM lists all components in your software, including libraries, dependencies, and their versions. When a new vulnerability is discovered, an SBOM lets you quickly determine which of your images are affected.

# Generate SBOM with Syft
syft myapp:latest -o spdx-json > sbom.spdx.json

# Generate SBOM with Trivy
trivy image --format spdx-json --output sbom.json myapp:latest

# Attach SBOM to image with Cosign
cosign attach sbom --sbom sbom.spdx.json myregistry/myapp:v1.0.0

# Scan SBOM for vulnerabilities
grype sbom:sbom.spdx.json

Store SBOMs alongside your images and keep them for the lifetime of the image. When Log4Shell or the next major vulnerability hits, you’ll be able to query your SBOMs to identify affected images within minutes rather than days.

Runtime Security

Securing images is essential but insufficient. Runtime security focuses on protecting containers while they execute, limiting what they can do even if compromised.

Running as Non-Root

By default, containers run as root. A vulnerability that allows code execution gives the attacker root privileges inside the container. Whilst namespaces and capabilities limit what root can do, defence in depth demands running as an unprivileged user.

FROM node:20-alpine

# Create a non-root user
RUN addgroup -g 1001 -S appgroup && \
    adduser -S appuser -u 1001 -G appgroup

WORKDIR /app

# Change ownership of application files
COPY --chown=appuser:appgroup . .

RUN npm ci --only=production

# Switch to non-root user
USER appuser

EXPOSE 3000

CMD ["node", "server.js"]

Some images require specific user IDs to match existing filesystem permissions or to integrate with orchestration platforms. Kubernetes, for example, can enforce that containers run as non-root:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 1001
    fsGroup: 1001
  containers:
    - name: app
      image: myapp:latest
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

Read-Only Root Filesystem

A read-only root filesystem prevents attackers from modifying binaries, dropping malware, or tampering with configuration. Any write attempts fail, limiting post-exploitation options.

# Run with read-only filesystem
docker run -d \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --tmpfs /var/run:rw,noexec,nosuid,size=16m \
  myapp:latest

The --tmpfs mounts provide writable temporary directories in memory for applications that need to write temporary files or PID files. The noexec option prevents execution of files in these directories.

Your Dockerfile should prepare for read-only operation:

FROM node:20-alpine

RUN adduser -S appuser

WORKDIR /app
COPY --chown=appuser . .
RUN npm ci --only=production

# Ensure no writes needed to root filesystem
# Pre-create any directories the app expects to exist
RUN mkdir -p /app/cache && chown appuser /app/cache

USER appuser

CMD ["node", "server.js"]

Resource Limits

Resource limits prevent denial-of-service attacks where a compromised container exhausts host resources. They also protect against accidental resource exhaustion from memory leaks or runaway processes.

docker run -d \
  --memory="256m" \
  --memory-swap="256m" \
  --memory-reservation="128m" \
  --cpus="0.5" \
  --cpu-shares=512 \
  --pids-limit=50 \
  --ulimit nofile=1024:1024 \
  --ulimit nproc=64:64 \
  myapp:latest

In Kubernetes, specify resource requests and limits:

apiVersion: v1
kind: Pod
metadata:
  name: resource-limited-pod
spec:
  containers:
    - name: app
      image: myapp:latest
      resources:
        requests:
          memory: "128Mi"
          cpu: "250m"
        limits:
          memory: "256Mi"
          cpu: "500m"

Network Policies

Network policies control which containers can communicate with each other and with external services. By default, all containers can communicate with all other containers—a compromised container can probe and attack anything on the network.

In Kubernetes, NetworkPolicy resources define allowed traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53

This policy allows the API pod to receive traffic only from the frontend application and monitoring namespace, and to send traffic only to the database and DNS.

For Docker without Kubernetes, use Docker networks to isolate containers:

# Create isolated networks
docker network create --internal backend-net
docker network create frontend-net

# Database only on internal network (no external access)
docker run -d --network backend-net --name db postgres:15

# API on both networks
docker run -d --network backend-net --network frontend-net --name api myapi:latest

# Frontend only on frontend network
docker run -d --network frontend-net -p 80:80 --name web nginx:alpine

Runtime Threat Detection

Runtime security tools monitor container behaviour and alert on or block suspicious activity. They detect attacks that static scanning can’t catch—exploits against zero-day vulnerabilities, malicious insider activity, or attacks using legitimate tools.

Falco

Falco is an open-source runtime security tool that monitors syscalls and alerts on suspicious behaviour.

# Example Falco rules
- rule: Terminal Shell in Container
  desc: Detect shell spawned in a container
  condition: >
    spawned_process and 
    container and 
    shell_procs and 
    proc.tty != 0
  output: >
    Shell spawned in container 
    (user=%user.name container=%container.name shell=%proc.name 
    parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

- rule: Write Below Binary Dir
  desc: Detect writes to binary directories
  condition: >
    open_write and 
    container and 
    bin_dir
  output: >
    Write to binary directory 
    (user=%user.name container=%container.name file=%fd.name)
  priority: CRITICAL
  tags: [container, filesystem, mitre_persistence]

- rule: Outbound Connection to Unusual Port
  desc: Detect outbound connections to non-standard ports
  condition: >
    outbound and 
    container and 
    not (fd.sport in (80, 443, 53, 8080, 8443, 5432, 6379, 3306))
  output: >
    Unusual outbound connection 
    (container=%container.name connection=%fd.name)
  priority: NOTICE
  tags: [container, network, mitre_exfiltration]

Deploy Falco in Kubernetes:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app: falco
  template:
    metadata:
      labels:
        app: falco
    spec:
      serviceAccountName: falco
      hostNetwork: true
      hostPID: true
      containers:
        - name: falco
          image: falcosecurity/falco:latest
          securityContext:
            privileged: true
          volumeMounts:
            - name: dev
              mountPath: /host/dev
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: etc
              mountPath: /host/etc
              readOnly: true
      volumes:
        - name: dev
          hostPath:
            path: /dev
        - name: proc
          hostPath:
            path: /proc
        - name: etc
          hostPath:
            path: /etc

Secret Management

Secrets in containers require careful handling. Environment variables, while convenient, can leak through logs, process listings, or debug endpoints. Build-time secrets must never persist in image layers. Runtime secrets should be injected securely and rotated regularly.

Build-Time Secrets

Docker BuildKit provides secure secret mounting during builds. Secrets are available only during the build step that mounts them and don’t persist in image layers.

# syntax=docker/dockerfile:1.4
FROM node:20-alpine

WORKDIR /app
COPY package*.json ./

# Mount npm token as secret - only available during this RUN
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc && \
    npm ci && \
    rm .npmrc

COPY . .
RUN npm run build

CMD ["node", "dist/server.js"]

Build with secrets:

# Pass secret from environment variable
export NPM_TOKEN="your-token-here"
docker buildx build --secret id=npm_token,env=NPM_TOKEN -t myapp:latest .

# Pass secret from file
echo "your-token-here" > npm_token.txt
docker buildx build --secret id=npm_token,src=npm_token.txt -t myapp:latest .

For SSH access to private repositories:

# syntax=docker/dockerfile:1.4
FROM golang:1.21 AS builder

WORKDIR /app

# Mount SSH agent for private repo access
RUN --mount=type=ssh \
    mkdir -p ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts && \
    go mod download

COPY . .
RUN go build -o myapp

FROM gcr.io/distroless/static
COPY --from=builder /app/myapp /
CMD ["/myapp"]

Build with SSH forwarding:

docker buildx build --ssh default -t myapp:latest .

Runtime Secrets

For runtime secrets, avoid environment variables where possible. Use mounted files or integrate with secret management systems.

Docker Secrets (Swarm)

# Create a secret
echo "supersecretpassword" | docker secret create db_password -

# Use in service
docker service create \
  --name api \
  --secret db_password \
  myapp:latest

Inside the container, the secret appears at /run/secrets/db_password.

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  database-password: "supersecretpassword"
  api-key: "sk-1234567890"
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: app-secrets

External Secret Stores

For production, integrate with external secret stores like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault:

# Using External Secrets Operator with AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    name: app-secrets
  data:
    - secretKey: database-password
      remoteRef:
        key: production/database
        property: password
    - secretKey: api-key
      remoteRef:
        key: production/api
        property: key

CI/CD Security Integration

Security must be automated into your CI/CD pipeline. Manual security reviews don’t scale, and security gates that slow down deployment get bypassed. Build security into the pipeline so it happens automatically on every build.

Comprehensive Security Pipeline

Here’s a GitHub Actions workflow that integrates vulnerability scanning, SBOM generation, and image signing:

name: Secure Container Build

on:
  push:
    branches: [main]
    tags: ['v*']
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  security-scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Run Trivy vulnerability scanner (filesystem)
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-fs-results.sarif'
      
      - name: Upload Trivy scan results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-fs-results.sarif'

  build-and-push:
    needs: security-scan
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write  # For keyless signing
      security-events: write
    
    outputs:
      digest: $
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: $/$
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=semver,pattern=
            type=semver,pattern=.
            type=sha
      
      - name: Build and push
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: $
          tags: $
          labels: $
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: true
          sbom: true
      
      - name: Run Trivy vulnerability scanner (image)
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: $/$:$
          format: 'sarif'
          output: 'trivy-image-results.sarif'
          severity: 'CRITICAL,HIGH'
        if: github.event_name != 'pull_request'
      
      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-image-results.sarif'
        if: github.event_name != 'pull_request'

  sign-image:
    needs: build-and-push
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    permissions:
      contents: read
      packages: write
      id-token: write
    
    steps:
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Sign the image with Cosign
        env:
          DIGEST: $
        run: |
          cosign sign --yes $/$@${DIGEST}

  generate-sbom:
    needs: build-and-push
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Install Syft
        uses: anchore/sbom-action/download-syft@v0
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Generate SBOM
        env:
          DIGEST: $
        run: |
          syft $/$@${DIGEST} -o spdx-json > sbom.spdx.json
      
      - name: Upload SBOM as artifact
        uses: actions/upload-artifact@v3
        with:
          name: sbom
          path: sbom.spdx.json

Security Gates

Define security thresholds that must be met before images can be deployed:

# Add to your build job
- name: Check vulnerability threshold
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: $/$:$
    format: 'table'
    exit-code: '1'  # Fail the build
    severity: 'CRITICAL'  # Only fail on critical vulnerabilities
    ignore-unfixed: true  # Ignore vulnerabilities without fixes

- name: Verify image signature before deployment
  run: |
    cosign verify \
      --certificate-identity-regexp="https://github.com/$/*" \
      --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
      $/$@$

Admission Control

In Kubernetes, use admission controllers to enforce security policies. Kyverno and OPA Gatekeeper can verify image signatures, enforce security contexts, and block non-compliant deployments.

# Kyverno policy requiring signed images
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/myorg/*"
---
# Policy enforcing security contexts
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-security-context
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Containers must run as non-root"
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
              - securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                      - ALL

Security Checklist

Here’s a comprehensive checklist for container security. Use it as a starting point and adapt it to your specific requirements:

Image Security

• Use minimal base images (distroless, Alpine, or scratch)
• Pin base image versions with digests, not just tags
• Scan images for vulnerabilities in CI/CD
• Generate and store SBOMs for all images
• Sign images with Cosign or similar
• Use multi-stage builds to exclude build tools
• Remove unnecessary packages and files
• Don’t install debugging tools in production images

Runtime Security

• Run containers as non-root users
• Drop all capabilities and add only what’s needed
• Use read-only root filesystems
• Set resource limits (memory, CPU, PIDs)
• Apply seccomp profiles
• Disable privilege escalation
• Use network policies to restrict communication
• Deploy runtime threat detection (Falco)

Secret Management

• Never commit secrets to version control
• Use BuildKit secrets for build-time credentials
• Mount secrets as files, not environment variables
• Integrate with external secret stores
• Rotate secrets regularly
• Audit secret access

CI/CD Security

• Scan code and dependencies before building
• Scan images after building
• Sign images in CI/CD pipeline
• Verify signatures before deployment
• Use admission controllers to enforce policies
• Implement security gates with severity thresholds

Infrastructure Security

• Keep container runtime updated
• Keep orchestration platform updated
• Use private registries with authentication
• Enable audit logging
• Monitor and alert on security events
• Regularly review and update security policies

Conclusion

Container security isn’t a single tool or practice—it’s a comprehensive approach that spans the entire lifecycle from base image selection through runtime monitoring. The shared kernel model means that container isolation, while effective, is not equivalent to the stronger isolation of virtual machines. Defence in depth is essential.

Start with the fundamentals: understand namespaces, cgroups, and capabilities. Choose minimal base images that reduce attack surface. Scan for vulnerabilities and generate SBOMs so you can respond quickly when new CVEs emerge. Run as non-root with minimal capabilities. Apply network policies to limit blast radius.

Automate security into your CI/CD pipeline so it happens consistently on every build. Sign your images to prove provenance. Use admission controllers to enforce policies at deployment time. Deploy runtime security tools to detect attacks that static analysis can’t catch.

Container security is an ongoing practice, not a one-time implementation. Vulnerabilities are discovered daily. Attackers evolve their techniques. Your security posture must evolve with them. Regular audits, updated policies, and continuous monitoring ensure your containerised applications remain protected as the threat landscape changes.

The investment in container security pays dividends in reduced risk, faster incident response, and confidence that your applications are protected by multiple layers of defence. You’ve built a foundation for secure containerised applications that will serve you well as your deployment grows.

For Azure-based platforms, the hub and spoke topology has emerged as the de facto standard for enterprise network design, and for good reason. It provides the perfect balance between isolation and connectivity, centralises security controls without creating bottlenecks, and scales elegantly as your organisation grows from a handful of applications to hundreds of workloads across multiple regions.

In this comprehensive guide, I’ll walk you through building a production-ready hub and spoke network topology in Azure using Terraform. We’ll explore the architectural principles that make this pattern so effective, implement the infrastructure with security best practices embedded at every layer, and address the operational considerations that separate toy examples from production-ready platforms.

Understanding the Hub and Spoke Pattern

Before we write a single line of Terraform, it’s worth understanding why this architectural pattern exists and what problems it solves. The hub and spoke topology isn’t just about drawing circles and lines on architecture diagrams, it’s a deliberate approach to managing complexity whilst maintaining security and operational efficiency.

The Problem with Flat Networks

In the early days of cloud adoption, many organisations built flat network topologies where every virtual network could communicate with every other virtual network through full-mesh peering. For three or four VNets, this works fine. By the time you reach ten VNets, you’re managing 45 peering connections. At twenty VNets, that number balloons to 190 peering connections.

Beyond the sheer management overhead, flat topologies create security challenges. Every VNet becomes a potential attack vector for every other VNet. Network security groups become impossibly complex as you try to maintain granular control over which applications can communicate. Audit trails become muddled as traffic flows directly between workloads without passing through central inspection points.

The Hub and Spoke Solution

The hub and spoke pattern solves these problems through centralisation. Instead of every VNet peering with every other VNet, spoke VNets peer only with a central hub VNet. The hub contains shared services; firewalls, VPN gateways, DNS servers, monitoring infrastructure, that all spokes can access. Traffic between spokes flows through the hub, where it can be inspected, logged, and controlled.

This topology dramatically reduces complexity. Twenty spoke VNets require only twenty peering connections to the hub, rather than 190 in a mesh topology. Security controls centralise in the hub, making them easier to maintain and audit. Network changes in one spoke don’t affect others unless you explicitly allow it.

The pattern also aligns beautifully with organisational structure. Each spoke can represent a different team, application, or environment. Development teams get their own isolated network space whilst platform engineering maintains central control over security policy, connectivity to on-premises networks, and shared infrastructure.

When to Use This Pattern

Hub and spoke topology makes sense when you have multiple workloads that need some degree of isolation from each other but also require shared services or connectivity to external networks. This describes most enterprise scenarios.

If you’re building a simple proof of concept with a single application in a single VNet, hub and spoke is overkill. But the moment you start thinking about multiple environments, multiple teams, or connections to on-premises networks, this pattern starts paying dividends. The initial investment in building the hub pays off quickly as you add spokes without increasing complexity.

Architectural Design Decisions

Every hub and spoke implementation requires making deliberate choices about how to structure the network, secure traffic, and handle connectivity. These decisions shape everything from IP address allocation to firewall rules to routing configuration.

IP Address Planning

IP address planning is one of those tasks that feels tedious until you get it wrong, at which point it becomes a nightmare. Once workloads are deployed and using IP addresses, changing them requires downtime, coordination, and significant risk.

The cardinal rule is to be generous with address space allocation. Azure supports RFC 1918 private address ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), and there’s no cost to reserving large blocks. I recommend allocating a /16 (65,536 addresses) for the hub and at least a /20 (4,096 addresses) for each spoke, with room to grow.

Within the hub, subdivide address space by function. Allocate a subnet for Azure Firewall, another for VPN Gateway, another for Azure Bastion, and so on. Each Azure service has specific subnet sizing requirements; Azure Firewall needs at least a /26, VPN Gateway needs at least a /27, Azure Bastion needs at least a /26. Factor in future growth when sizing these subnets.

For spoke VNets, organise subnets by tier or function. Web tier gets a subnet, application tier gets another, data tier gets a third. Keep subnet sizes consistent across spokes where possible—it makes automation easier and helps with capacity planning.

Document your IP address allocation scheme from the start. Future you, six months from now, will thank present you for writing down which address ranges are allocated to which environments and teams.

Hub VNet Design

The hub VNet serves as the central point of connectivity for your entire network topology. It needs to accommodate several distinct functions, each with its own subnet requirements.

Azure Firewall sits at the heart of the hub, inspecting all traffic between spokes and controlling outbound internet access. It requires a dedicated subnet named AzureFirewallSubnet with a minimum size of /26, though /24 gives you more room for scale.

If you’re connecting to on-premises networks via VPN or ExpressRoute, you’ll need a Gateway subnet. VPN Gateway and ExpressRoute Gateway both require a subnet named GatewaySubnet with a minimum size of /27, though /26 is recommended for production deployments.

Azure Bastion provides secure RDP and SSH access to virtual machines without exposing them to the public internet. It needs a subnet named AzureBastionSubnet with a minimum size of /26.

For shared services like jump boxes, monitoring infrastructure, or centralised DNS servers, create additional subnets sized appropriately for your needs. A /27 or /28 often suffices for these management workloads.

Spoke VNet Design

Spoke VNets contain your actual workloads; the applications, databases, and services that deliver value to your organisation. Design them with isolation and security in mind.

Each spoke should map to a clear organisational or technical boundary. You might create spokes per environment (development, staging, production), per team (platform, data, mobile), per application tier (frontend, backend, data), or some combination. The key is consistency; establish a pattern and stick to it.

Within each spoke, use subnets to create security boundaries. Don’t put web servers and databases in the same subnet. Subnet boundaries allow you to apply network security groups that control which traffic is allowed. A compromised web server in its own subnet can be prevented from accessing database servers in a different subnet.

Consider creating separate subnets for private endpoints if you’re using Azure PaaS services like Storage Accounts or SQL Databases with private connectivity. This makes it easier to manage DNS resolution and network security group rules for these endpoints.

Routing Architecture

In a hub and spoke topology, routing determines how traffic flows between spokes. By default, spoke-to-spoke traffic doesn’t work even with peering configured; Azure VNet peering is non-transitive. If Spoke A peers with the Hub, and Spoke B peers with the Hub, traffic from Spoke A cannot reach Spoke B without additional configuration.

This is actually a security feature. It means you explicitly control which spokes can communicate, rather than allowing all-to-all communication by default.

To enable spoke-to-spoke communication, you have two options. The simpler approach uses Azure Firewall as a router. Configure user-defined routes (UDRs) in each spoke that send traffic destined for other spoke address ranges to the Azure Firewall. The firewall inspects the traffic and forwards it to the destination spoke.

The alternative approach deploys a network virtual appliance (NVA) in the hub for routing, but for most scenarios, Azure Firewall provides sufficient routing capabilities alongside its security functions. Using a single service for both simplifies management and reduces cost.

Connectivity Options

Your hub and spoke topology likely needs connectivity beyond Azure. On-premises datacentres, remote offices, and mobile workers all need secure access to cloud resources.

For site-to-site connectivity to on-premises networks, you’ll choose between VPN Gateway and ExpressRoute. VPN Gateway provides encrypted IPsec tunnels over the public internet and supports throughput up to 10 Gbps with the VpnGw5 SKU. It’s cost-effective and suitable for most scenarios.

ExpressRoute provides private connectivity that doesn’t traverse the public internet, with dedicated bandwidth from 50 Mbps to 100 Gbps. It costs more but provides better performance, reliability, and security for mission-critical workloads. Many organisations use both; ExpressRoute for primary connectivity and VPN Gateway as a failover path.

For remote user access, Azure Bastion provides secure RDP and SSH connectivity without exposing virtual machines to the internet. It eliminates the need for jump boxes with public IP addresses and provides audit logging of all remote access sessions.

Visual Architecture

To help visualise how all these components fit together, here’s a diagram showing a typical hub and spoke topology with three spoke VNets:

graph TB
    subgraph Internet["Internet"]
        OnPrem["On-Premises Network
192.168.0.0/16"]
        Users["Remote Users"]
    end

    subgraph Hub["Hub VNet
10.0.0.0/16"]
        subgraph FirewallSubnet["AzureFirewallSubnet
10.0.0.0/24"]
            AFW["Azure Firewall
10.0.0.4"]
        end
        
        subgraph GatewaySubnet["GatewaySubnet
10.0.1.0/24"]
            VPN["VPN Gateway"]
        end
        
        subgraph BastionSubnet["AzureBastionSubnet
10.0.2.0/24"]
            Bastion["Azure Bastion"]
        end
        
        subgraph ManagementSubnet["Management Subnet
10.0.3.0/24"]
            Monitor["Monitoring
Infrastructure"]
        end
    end

    subgraph Spoke1["Production Spoke
10.1.0.0/16"]
        subgraph WebSubnet1["Web Subnet
10.1.1.0/24"]
            Web1["Web Servers"]
        end
        subgraph AppSubnet1["App Subnet
10.1.2.0/24"]
            App1["Application
Servers"]
        end
        subgraph DataSubnet1["Data Subnet
10.1.3.0/24"]
            DB1["Databases"]
        end
        RT1["Route Table"]
    end

    subgraph Spoke2["Development Spoke
10.2.0.0/16"]
        subgraph WorkloadSubnet2["Workload Subnet
10.2.1.0/24"]
            Workload2["Development
Workloads"]
        end
        RT2["Route Table"]
    end

    subgraph Spoke3["Shared Services Spoke
10.3.0.0/16"]
        subgraph ServicesSubnet3["Services Subnet
10.3.1.0/24"]
            DNS["DNS Servers"]
            AD["Active Directory"]
        end
        RT3["Route Table"]
    end

    OnPrem -.->|"VPN Tunnel
IPsec"| VPN
    Users -.->|"RDP/SSH"| Bastion
    
    VPN -->|"Peering"| AFW
    Bastion -.->|"Secure Access"| Web1
    Bastion -.->|"Secure Access"| Workload2
    
    AFW <-->|"VNet Peering
Allow Gateway Transit"| Spoke1
    AFW <-->|"VNet Peering
Allow Gateway Transit"| Spoke2
    AFW <-->|"VNet Peering
Allow Gateway Transit"| Spoke3
    
    RT1 -.->|"0.0.0.0/0 → Firewall
10.2.0.0/16 → Firewall
10.3.0.0/16 → Firewall"| AFW
    RT2 -.->|"0.0.0.0/0 → Firewall
10.1.0.0/16 → Firewall
10.3.0.0/16 → Firewall"| AFW
    RT3 -.->|"0.0.0.0/0 → Firewall
10.1.0.0/16 → Firewall
10.2.0.0/16 → Firewall"| AFW
    
    Web1 -->|"App Traffic"| App1
    App1 -->|"Database
Queries"| DB1
    
    Workload2 -.->|"Via Firewall"| DNS
    App1 -.->|"Via Firewall"| DNS
    
    AFW -->|"Inspect & Log"| Internet
    
    Monitor -.->|"Collect Logs"| AFW
    Monitor -.->|"Collect Logs"| VPN
    Monitor -.->|"Collect Logs"| Bastion

    classDef hubStyle fill:#0078d4,stroke:#003d7a,stroke-width:2px,color:#fff
    classDef spokeStyle fill:#50e6ff,stroke:#0078d4,stroke-width:2px,color:#000
    classDef securityStyle fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff
    classDef gatewayStyle fill:#69db7c,stroke:#2b8a3e,stroke-width:2px,color:#000
    
    class AFW securityStyle
    class VPN,Bastion gatewayStyle
    class Spoke1,Spoke2,Spoke3 spokeStyle
```

Click the diagram to view fullscreen

This diagram illustrates several key architectural points. The hub sits at the centre with four distinct subnets, each serving a specific purpose. Azure Firewall acts as the central inspection point for all inter-spoke traffic and internet-bound traffic. VPN Gateway provides the encrypted tunnel to on-premises networks, whilst Azure Bastion offers secure administrative access without exposing VMs to the internet.

Each spoke VNet peers directly with the hub but not with other spokes. The route tables in each spoke contain user-defined routes that send traffic destined for other spokes through the Azure Firewall’s private IP address. This ensures all spoke-to-spoke communication flows through the firewall where it can be inspected, logged, and controlled by centralised firewall policies.

Notice how the production spoke uses a traditional three-tier subnet design (web, application, data), whilst the development spoke has a simpler structure with a single workload subnet. The shared services spoke contains infrastructure used by multiple teams, like DNS servers and Active Directory. This flexibility in spoke design is one of the pattern’s strengths—each spoke can be structured according to its specific requirements whilst maintaining consistent connectivity and security through the hub.

Infrastructure as Code with Terraform

Now let’s translate these architectural principles into actual infrastructure. We’ll build this incrementally, starting with the hub and progressively adding spokes and connectivity.

Foundation and Variables

Start by defining the variables that will drive your configuration. Create variables.tf:

variable "environment" {
  description = "Environment name (e.g., prod, staging, dev)"
  type        = string
}

variable "location" {
  description = "Primary Azure region"
  type        = string
  default     = "uksouth"
}

variable "organisation" {
  description = "Organisation name for resource naming"
  type        = string
}

variable "hub_vnet_address_space" {
  description = "Address space for hub VNet"
  type        = list(string)
  default     = ["10.0.0.0/16"]
}

variable "spoke_vnets" {
  description = "Map of spoke VNets to create"
  type = map(object({
    address_space = list(string)
    subnets = map(object({
      address_prefix = string
      service_endpoints = optional(list(string), [])
      delegation = optional(object({
        name = string
        service_delegation = object({
          name    = string
          actions = optional(list(string), [])
        })
      }))
    }))
  }))
  default = {}
}

variable "enable_vpn_gateway" {
  description = "Whether to deploy VPN Gateway in hub"
  type        = bool
  default     = true
}

variable "enable_bastion" {
  description = "Whether to deploy Azure Bastion in hub"
  type        = bool
  default     = true
}

variable "on_premises_address_spaces" {
  description = "Address spaces of on-premises networks for VPN"
  type        = list(string)
  default     = []
}

variable "tags" {
  description = "Tags to apply to all resources"
  type        = map(string)
  default     = {}
}

Create locals.tf for computed values:

locals {
  # Common tags applied to all resources
  common_tags = merge(
    var.tags,
    {
      environment = var.environment
      managed_by  = "terraform"
      pattern     = "hub-and-spoke"
    }
  )

  # Resource naming convention
  hub_name = "hub-${var.environment}-${var.location}"
  
  # Hub subnet configuration
  hub_subnets = {
    firewall = {
      name           = "AzureFirewallSubnet"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 0)
    }
    gateway = {
      name           = "GatewaySubnet"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 1)
    }
    bastion = {
      name           = "AzureBastionSubnet"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 2)
    }
    management = {
      name           = "snet-management"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 3)
    }
  }
}

Hub Virtual Network

Create the hub VNet with all its subnets. Create hub.tf:

# Resource group for hub networking
resource "azurerm_resource_group" "hub" {
  name     = "rg-network-${local.hub_name}"
  location = var.location
  tags     = local.common_tags
}

# Hub virtual network
resource "azurerm_virtual_network" "hub" {
  name                = "vnet-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  address_space       = var.hub_vnet_address_space

  tags = local.common_tags
}

# Hub subnets
resource "azurerm_subnet" "hub_firewall" {
  name                 = local.hub_subnets.firewall.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.firewall.address_prefix]
}

resource "azurerm_subnet" "hub_gateway" {
  count = var.enable_vpn_gateway ? 1 : 0

  name                 = local.hub_subnets.gateway.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.gateway.address_prefix]
}

resource "azurerm_subnet" "hub_bastion" {
  count = var.enable_bastion ? 1 : 0

  name                 = local.hub_subnets.bastion.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.bastion.address_prefix]
}

resource "azurerm_subnet" "hub_management" {
  name                 = local.hub_subnets.management.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.management.address_prefix]

  service_endpoints = [
    "Microsoft.Storage",
    "Microsoft.KeyVault"
  ]
}

# Network security group for management subnet
resource "azurerm_network_security_group" "hub_management" {
  name                = "nsg-${local.hub_name}-management"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name

  tags = local.common_tags
}

# NSG association
resource "azurerm_subnet_network_security_group_association" "hub_management" {
  subnet_id                 = azurerm_subnet.hub_management.id
  network_security_group_id = azurerm_network_security_group.hub_management.id
}

Azure Firewall

Azure Firewall provides the central security control point for the topology. Create firewall.tf:

# Public IP for Azure Firewall
resource "azurerm_public_ip" "firewall" {
  name                = "pip-firewall-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  allocation_method   = "Static"
  sku                 = "Standard"
  zones               = ["1", "2", "3"]

  tags = local.common_tags
}

# Azure Firewall
resource "azurerm_firewall" "hub" {
  name                = "afw-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  sku_name            = "AZFW_VNet"
  sku_tier            = "Standard"
  firewall_policy_id  = azurerm_firewall_policy.hub.id
  zones               = ["1", "2", "3"]

  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.hub_firewall.id
    public_ip_address_id = azurerm_public_ip.firewall.id
  }

  tags = local.common_tags
}

# Firewall Policy
resource "azurerm_firewall_policy" "hub" {
  name                = "afwp-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  sku                 = "Standard"
  
  threat_intelligence_mode = "Alert"

  dns {
    proxy_enabled = true
  }

  intrusion_detection {
    mode = "Alert"
  }

  tags = local.common_tags
}

# Firewall Policy Rule Collection Group - Network Rules
resource "azurerm_firewall_policy_rule_collection_group" "network_rules" {
  name               = "network-rules"
  firewall_policy_id = azurerm_firewall_policy.hub.id
  priority           = 100

  network_rule_collection {
    name     = "allow-spoke-to-spoke"
    priority = 100
    action   = "Allow"

    rule {
      name                  = "allow-all-spoke-to-spoke"
      protocols             = ["Any"]
      source_addresses      = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_addresses = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_ports     = ["*"]
    }
  }

  network_rule_collection {
    name     = "allow-dns"
    priority = 110
    action   = "Allow"

    rule {
      name                  = "allow-dns-outbound"
      protocols             = ["UDP"]
      source_addresses      = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_addresses = ["*"]
      destination_ports     = ["53"]
    }
  }
}

# Firewall Policy Rule Collection Group - Application Rules
resource "azurerm_firewall_policy_rule_collection_group" "application_rules" {
  name               = "application-rules"
  firewall_policy_id = azurerm_firewall_policy.hub.id
  priority           = 200

  application_rule_collection {
    name     = "allow-azure-services"
    priority = 100
    action   = "Allow"

    rule {
      name = "allow-azure-management"
      protocols {
        type = "Https"
        port = 443
      }
      source_addresses = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_fqdns = [
        "*.azure.com",
        "*.microsoft.com",
        "*.windows.net",
        "*.azure-automation.net"
      ]
    }
  }

  application_rule_collection {
    name     = "allow-ubuntu-updates"
    priority = 110
    action   = "Allow"

    rule {
      name = "allow-apt-repositories"
      protocols {
        type = "Http"
        port = 80
      }
      protocols {
        type = "Https"
        port = 443
      }
      source_addresses = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_fqdns = [
        "*.ubuntu.com",
        "*.canonical.com"
      ]
    }
  }
}