This post documents the symptoms, the root cause, and the exact steps required to recover a cluster stuck in this half‑installed, half‑removed state.

The Setup

This cluster was:

• Running AKS with the old AAD Pod Identity Helm chart still installed
• Missing MIC (Managed Identity Controller)
• Still running NMI (Node Managed Identity) on every node
• Running Flux v2 with image automation enabled
• Using ACR with a node‑assigned managed identity

This combination is more common than you’d think—especially in clusters upgraded over several years.

The Symptoms

Flux’s image-reflector-controller began failing with IMDS authentication errors:

failed to configure authentication options: failed to create provider access token for the controller: ManagedIdentityCredential: context deadline exceeded

NMI logs showed:

failed to get matching identities for pod: flux-system/image-reflector-controller…

Flux could not:

• Query ACR
• Populate its tag database
• Resolve image policies

Image automation was effectively dead.

Diagnosis: The Smoking Gun

Running kubectl get pods -n kube-system | grep nmi revealed dozens of NMI pods, all running for hundreds of days.

kubectl get crd | grep aadpodidentity revealed AAD Pod Identity CRDs still installed.

But running kubectl get pods -n kube-system | grep mic showed no MIC pods at all.

This is the broken state:

• NMI intercepts IMDS calls
• MIC is missing, so identities are never assigned
• Every IMDS call from a pod fails
• Flux cannot authenticate to ACR

This is where the cluster was stuck.

Why This Breaks Flux

Flux’s controllers authenticate to ACR using the node’s managed identity via IMDS.

But NMI intercepts IMDS calls and tries to look up a pod’s assigned identity.

With MIC missing, NMI always returns:

no AzureAssignedIdentity found

Flux never reaches IMDS → never reaches ACR → never sees image tags.

The Fix: AzurePodIdentityException

The solution is to tell NMI:

“Do not intercept IMDS calls for Flux pods.”

This is done using an AzurePodIdentityException.

But here’s the twist:
AAD Pod Identity has multiple CRD schemas depending on the version, and this cluster was running a very old one.

Attempts using modern fields like:

spec:
  podSelector:

or

spec:
  PodLabels:

were rejected.

The correct schema, discovered by inspecting existing exceptions (kubectl get azurepodidentityexception -n flux-system -o yaml), was:

spec:
  podLabels:
    <label>: <value>

I discovered a couple of existing exception for Flux that didn’t cover the label used by the image-reflector-controller pod:

spec:
  podLabels:
    app.kubernetes.io/name: flux-extension

Creating the Correct Exception

I inspected the labels of the pods in the flux-system namespace:

kubectl get pods -n flux-system --show-labels

Flux controllers all share this label:

app.kubernetes.io/name=microsoft.flux

So the correct exception is:

apiVersion: aadpodidentity.k8s.io/v1
kind: AzurePodIdentityException
metadata:
  name: flux-system-exception
  namespace: flux-system
spec:
  podLabels:
    app.kubernetes.io/name: microsoft.flux

Apply it:

kubectl apply -f flux-system-exception.yaml

Then restart Flux:

kubectl rollout restart deploy -n flux-system

Verifying the Fix

Check NMI logs:

kubectl logs -n kube-system -l app.kubernetes.io/component=nmi --tail=200

Before the fix:

failed to get matching identities for pod flux-system/image-reflector-controller…

After the fix no errors are shown.

Flux logs then showed healthy behaviour:

successful scan: found 1 tags

Latest image tag … resolved to prod-8552309f0-20260512T090128

Image automation was back.

Final Thoughts

This issue is obscure, poorly documented, and easy to miss. If you’re running Flux on AKS and see IMDS or ACR authentication failures, check for:

• NMI pods still running
• MIC missing
• Old AAD Pod Identity CRDs
• Missing or incorrect AzurePodIdentityException objects

Hopefully this post saves someone else the hours of digging it took to unravel this.

This post is a practical reference for:

• Signing OCI container images with cosign (Sigstore)
• Generating and attaching attestations (including SBOM-related examples)
• Storing signatures/attestations alongside images in Azure Container Registry (ACR)
• Understanding how the OCI Referrers API organises “artifacts about artifacts”
• Briefly comparing Cosign to Notation (Notary Project)

Key concepts (what are we actually storing?)

When people say “sign the image” they often mean a few related, but distinct things.

Signatures

A signature is a statement like:

“I (an identity) signed this immutable image digest.”

The important part is that you sign the image digest (@sha256:...), not the tag (:latest), because tags are mutable.

Attestations

An attestation is a signed statement about the image (or about another artifact attached to the image). Examples:

• Provenance: where the build ran, what sources were used, what inputs were consumed
• SBOM: a machine-readable list of packages/components contained in the image
• Vulnerability scan results (careful: these go stale quickly)

Cosign signs attestations using DSSE envelopes and commonly uses the in-toto attestation format.

SBOMs

An SBOM is a document — it’s valuable when you:

• Generate it consistently
• Store it alongside the exact digest you shipped
• Sign it (so people can trust its origin)
• Make it discoverable by automation

Why sign images and generate attestations?

If you’re already scanning images, signatures and attestations provide the missing chain of custody.

• Integrity: detect tampering (a signature ties to an immutable digest)
• Authenticity: verify “this came from our pipeline/team”, not just “it exists in our registry”
• Policy enforcement: gate deployments on verified signatures/attestations (Kubernetes admission, CI/CD checks)
• Auditability: prove what was deployed and when, and how it was built
• Better incident response: quickly answer “which builds include vulnerable component X?” when you have SBOMs attached to digests

Where do signatures/attestations live? OCI Artifacts + the Referrers API

Modern container signing doesn’t embed signatures “inside” an image. Instead, registries store additional OCI artifacts that refer to the image.

The OCI Distribution Spec defines a Referrers API (OCI v1.1) that lets a client ask:

“List all artifacts that refer to this subject digest.”

That makes signatures, SBOMs, provenance, scan reports, etc. first-class, discoverable artifacts associated with an image.

Azure Container Registry supports storing and discovering these supply-chain artifacts. Tools like ORAS can show the attached artifact graph, and ACR supports the OCI Referrers API in most scenarios.

Note: some environments may fall back to the older “referrers tag schema” approach instead of the Referrers API. For example, ACR’s documentation notes that some features (notably customer-managed key encrypted registries) may require fallback behavior.

Cosign vs Notation (quick comparison)

Both Cosign and Notation ultimately store “signatures as OCI artifacts” in registries, but they’re optimised for different workflows.

Cosign (Sigstore)

• Excellent developer experience for keyless signing using OIDC identities (Fulcio certificates) and transparency logging (Rekor)
• Strong support for attestations (provenance, SBOM predicates, etc.)
• Works well in modern CI/CD environments where identity is provided by an OIDC issuer (GitHub Actions, etc.)

Notation (Notary Project)

• Industry standardisation effort for OCI signing with an emphasis on X.509 / trust policies
• Strong fit for enterprise key management, including KMS/HSM-backed keys (for example via an Azure Key Vault plugin)
• Focused primarily on signing/verifying artifacts; attestations are commonly handled via separate OCI artifacts and tooling

If you’re building a modern “provenance + SBOM + policy” workflow, Cosign is often the fastest path. If you need a standardised signature format and strong enterprise policy/trust-store modeling (and/or a vendor-supported path), Notation is worth serious consideration.

Walkthrough: build, push, sign, and attach attestations in ACR

This walkthrough uses:

• Azure Container Registry (ACR)
• cosign for signing and attestations
• oras to discover and reason about the artifact graph (Referrers API)

Prerequisites

• Azure CLI (az) authenticated to your subscription
• ACR with permissions to push images and artifacts
• Tools:
  • cosign
  • oras
  • jq (optional, but helpful)
  • SBOM generator such as syft (optional)

On macOS you can typically install these with Homebrew (adjust as needed):

brew install cosign oras jq syft

1) Create an ACR and sign in

az login

export LOCATION=uksouth
export RG=rg-acr-signing-demo
export ACR_NAME=<your-unique-acr-name>

az group create -n "$RG" -l "$LOCATION"
az acr create -g "$RG" -n "$ACR_NAME" --sku Standard

export REGISTRY="$ACR_NAME.azurecr.io"
az acr login -n "$ACR_NAME"

2) Build and push an image (and capture the digest)

You can build locally with Docker and push, but for a clean “works anywhere” demo, ACR Tasks is handy.

export REPO=demo/net-monitor
export TAG=v1

# Example source repo used in Microsoft docs; swap this for your own Dockerfile repo.
export IMAGE_SOURCE=https://github.com/wabbit-networks/net-monitor.git#main

DIGEST=$(az acr build \
	-r "$ACR_NAME" \
	-t "$REGISTRY/${REPO}:$TAG" \
	"$IMAGE_SOURCE" \
	--no-logs \
	--query "outputImages[0].digest" -o tsv)

export IMAGE_DIGEST="$REGISTRY/${REPO}@$DIGEST"
echo "$IMAGE_DIGEST"

From here on, use $IMAGE_DIGEST (digest reference) for signing and attestations.

3) Sign the image with Cosign (keyless)

Keyless signing is Cosign’s default “batteries included” path.

cosign sign --yes "$IMAGE_DIGEST"

What to expect:

• A browser-based OIDC flow (unless you’re already in a CI environment with an OIDC token)
• A short-lived signing certificate issued by Sigstore’s Fulcio
• A transparency log entry in Rekor (by default)
• A signature stored in your registry as an OCI artifact referring to the image

4) Verify the Cosign signature

Verification should be part of your CI and/or cluster admission policy. The important idea is: verify not only “is it signed?” but “is it signed by the identity we trust?”.

For interactive keyless signing, the OIDC issuer is commonly:

OIDC_ISSUER="https://oauth2.sigstore.dev/auth"

And the identity is typically the OIDC subject (often your email). Example:

cosign verify \
	--certificate-identity "you@example.com" \
	--certificate-oidc-issuer "$OIDC_ISSUER" \
	"$IMAGE_DIGEST"

In CI (for example GitHub Actions), you’d instead verify against that workflow identity and issuer (for example https://token.actions.githubusercontent.com), often using the --certificate-identity-regexp flags to match your org/repo/workflow.

5) Generate an SBOM and attach it as an attestation (Cosign)

One straightforward pattern is:

1) Generate an SBOM file 2) Attest to it (sign a statement that includes it)

Generate an SBOM with Syft (SPDX JSON example):

syft packages "$IMAGE_DIGEST" -o spdx-json=sbom.spdx.json

Create an attestation for that SBOM:

cosign attest --yes \
	--predicate sbom.spdx.json \
	--type spdxjson \
	"$IMAGE_DIGEST"

Verify the attestation:

cosign verify-attestation \
	--certificate-identity "you@example.com" \
	--certificate-oidc-issuer "$OIDC_ISSUER" \
	--type spdxjson \
	"$IMAGE_DIGEST" \
	| jq -r '.[0].payload' \
	| base64 --decode

6) Discover what’s attached (OCI Referrers) with ORAS

This is the piece most people miss: once signatures/SBOMs/attestations are stored as OCI artifacts, you can query the graph.

oras discover -o tree "$IMAGE_DIGEST"

You should see the subject image at the top, with child artifact types beneath it (signatures, attestations, and any other artifacts you attach).

Alternative pattern: store the SBOM as an OCI artifact, then sign that

Sometimes you want the SBOM itself stored as a first-class artifact in the registry (so it can be pulled directly), and then you sign/attest to that artifact. OCI referrers support deep graphs: image → SBOM → signature.

Attach an SBOM file to the image as a referenced OCI artifact:

oras attach "$IMAGE_DIGEST" \
	--artifact-type sbom/example \
	./sbom.spdx.json:application/json

Discover again:

oras discover -o tree "$IMAGE_DIGEST"

This pattern is powerful when you want:

• direct pull of SBOMs (oras pull), separate from verification tooling
• to sign the SBOM artifact independently (for example with Notation)

Promoting images with their signatures/attestations

If you promote images across registries/environments, you should promote the artifact graph, not only the image manifest.

With ORAS you can copy an image and its referrers:

export TARGET_REPO="$REGISTRY/staging/$REPO"
oras copy -r "$IMAGE_DIGEST" "$TARGET_REPO:$TAG"

Then:

oras discover -o tree "$TARGET_REPO:$TAG"

Notation (brief practical example in Azure)

If you’re evaluating Notation in Azure, Microsoft provides a walkthrough that signs images in ACR using keys held in Azure Key Vault. The high-level flow looks like:

1) Install Notation + the Azure Key Vault plugin 2) Build/push image and sign by digest 3) Add certs to a trust store and create a trust policy 4) Verify

Signing example (Key Vault plugin; simplified):

notation sign \
	--signature-format cose \
	--id "$KEY_ID" \
	--plugin azure-kv \
	--plugin-config self_signed=true \
	"$IMAGE_DIGEST"

And list signatures:

notation ls "$IMAGE_DIGEST"

An important nuance called out in Microsoft’s guidance: Notation can store signatures using the referrers tag schema by default, and can also use the OCI Referrers API depending on configuration/registry capabilities.

Common pitfalls

• Always sign by digest (@sha256:...), not by tag.
• Verification must include identity constraints, not only “is signed?”.
• Plan where your trust roots live: developer keyless identity, CI workload identity, or enterprise PKI (Notation) are different trust models.
• Think about artifact lifecycle: deleting an image doesn’t always automatically delete every related artifact in every registry implementation.
• Referrers API vs tag schema: some environments may require fallback behavior; test your exact ACR configuration (including encryption features) if you’re standardising on Referrers API workflows.

Suggested “good default” pipeline shape

If you’re starting from scratch, a practical baseline is:

1) Build image 2) Push image 3) Generate SBOM 4) Create provenance (if your build system supports it) 5) Sign image digest 6) Attest SBOM/provenance to image digest 7) Enforce verification at deploy time (admission policy)

If you want to go further, add:

• environment-specific signatures (promotion gates)
• oras copy -r promotion of artifact graphs
• key management hardening (HSM/KMS), timestamping, and policy-as-code

What is Peep?

Peep is a system monitoring application for macOS that displays real-time information about your computer’s performance. Think of it as a modern, visually appealing alternative to Activity Monitor. It features:

• Real-time CPU and memory monitoring with animated gauges
• Network and disk I/O tracking with historical charts
• Process management with tree view and thread filtering
• Battery status with time remaining estimates
• Machine information including model name and macOS version

The tech stack combines Electron for the desktop shell, React and TypeScript for the UI, and Rust via Neon bindings for high-performance native system access.

Vibe Coding: A New Development Paradigm

“Vibe coding” is an emerging term for a development style where you describe what you want in natural language and iterate rapidly with AI assistance. Rather than meticulously planning every implementation detail upfront, you:

1. Describe the desired outcome - “Add battery information to the header”
2. Review the generated solution - AI provides working code
3. Refine through conversation - “The icon isn’t showing correctly for discharging state”
4. Iterate until satisfied - Continue the dialogue until it works perfectly

This approach flips traditional development on its head. Instead of spending hours reading documentation and debugging, you spend time describing and refining. The AI handles the boilerplate, the API lookups, and often catches edge cases you hadn’t considered.

The Development Journey

Starting Point: Architecture Decisions

The project began with a clear vision: a system monitor that felt native to macOS while being built with web technologies. The architecture emerged through conversation:

Me: “I want to build a system monitor for macOS using Electron”

Copilot: Suggested the full stack; Electron for the shell, React for the UI, and importantly, recommended Rust with Neon bindings for the native module rather than pure Node.js. This was crucial for performance when polling system metrics every few seconds.

The initial scaffolding; webpack config, TypeScript setup, IPC handlers, Rust/Neon integration, was generated in minutes. Tasks that would typically take half a day of configuration wrestling were done before my coffee got cold.

Copilot opens a pull request with a thorough explanation of the work that has been done. At this point I can pull the branch to my machine and do some QA.

The initial version created by Copilot looked like so:

If I find any bugs or want to make some minor improvements I can add a comment to the pull request.

glenthomas commented

@copilot the nodejs project is trying to import the Rust binary and is failing with error:

Failed to load native module: Error: Cannot find module ‘../../native/index.node’

Require stack:

/Users/glen.thomas/git/peep/dist/main.js

at Module._resolveFilename (node:internal/modules/cjs/loader:1390:15)

Copilot add the eyes emoji (👀) to to my comment to acknowledge my request and gets straight to work in a new session. A new comment is added to the pull request to let me know when the next stage of the work has been completed.

Copilot AI commented

Fixed in commit e1003c2. The issue was that the native module path was using a hardcoded relative require path that didn’t resolve correctly at runtime.

Changes made:

1. Updated src/main/main.ts to use path.join(__dirname, ‘..’, ‘native’, ‘index.node’) for proper path resolution
2. Added @types/node and @types/electron as dev dependencies (required for TypeScript compilation)
3. Updated build script to use npx cargo-cp-artifact to ensure the tool is available
4. Enhanced README troubleshooting section

The native module now loads correctly. From dist/main.js, it resolves to /native/index.node. Verified with test that shows the module loads successfully and can retrieve system info.

Feature Development: The Tree View Example

One of the more complex features was adding a process tree view that shows parent-child relationships between processes. Here’s how that conversation went:

Me: “Could we have an option to view process parent-child relationships?”

Within moments, I had:

• A buildProcessTree() function that constructs a hierarchy from the flat process list
• A renderProcessTree() function with proper indentation and expand/collapse
• Toggle buttons for switching between list and tree views
• State management for tracking expanded nodes

The implementation used the ppid (parent process ID) field that was already being collected by the Rust module. Copilot recognised the data was available and used it appropriately—something that would have required me to trace through the codebase manually.

Debugging: The Battery Icon Bug

Not everything worked perfectly on the first try, and that’s where the iterative nature of vibe coding shines. After adding battery status to the header, I noticed the icon wasn’t correct:

Me: “The battery state icon is not showing the correct icon. When the battery state is ‘discharging’, it shows the charging icon.”

Copilot identified the issue immediately: the getStateIcon() function was using includes() to match states, so “discharging” was matching the “charging” case first. The fix was simple—use exact matching with === and startsWith() instead.

This kind of bug could easily waste 20 minutes of console.log debugging. Instead, describing the symptom led directly to the fix.

The UI Polish Phase

As the core functionality stabilised, I shifted to UI refinements. This is where vibe coding particularly excels - making small adjustments is as easy as asking:

• “The columns, view type and show threads controls are stacked vertically, can they be stacked horizontally?” → Segmented button groups
• “On the CPU monitor and Memory monitor there is a bit of a vertical gap” → SVG viewport adjustments
• “Can we decrease the size of the gauges a little? Maybe 20%” → Proportional scaling of all gauge dimensions
• “The battery info is not properly centered in the header” → CSS Grid with 1fr auto 1fr

Each of these would typically involve researching CSS properties, testing various approaches, and iterating. With Copilot, I described the visual problem and received targeted solutions.

Native Integration Challenges

Some features required deeper system integration. When I asked about displaying the machine model name, the response included:

1. Using sysctl to read hw.model (e.g., “MacBookPro18,1”)
2. A mapping table translating hardware identifiers to friendly names
3. Integration with the existing get_os_info() Rust function

Similarly, adding the macOS marketing name (Sonoma, Sequoia, etc.) required mapping Darwin version numbers to release names—knowledge that would require research but was immediately available through Copilot.

When AI Knowledge Falls Short

It wasn’t all smooth sailing. One notable friction point came when working with the Rust sysinfo crate. Copilot confidently suggested methods and function calls that simply didn’t exist in the current version of the library.

For example, when implementing disk I/O monitoring per process, the AI suggested using methods that had either been renamed, deprecated, or never existed in the version I was using (0.37.x). The code would compile with errors like “method not found” or “no field named X on type Y”.

This is a fundamental limitation of AI coding assistants: their training data has a cutoff date, and libraries evolve. The sysinfo crate in particular has undergone significant API changes between versions, with methods being renamed and structs being reorganised.

The solution? I had to do things the old-fashioned way—pull up the sysinfo documentation on docs.rs, find the correct method names for my version, and either correct the AI’s suggestions or implement the fix myself. In some cases, I’d share the compiler error and the relevant documentation snippet, and Copilot would adjust. In others, it was faster to just fix it manually.

This experience reinforced an important lesson: AI assistants are collaborators, not oracles. They’re incredibly useful for getting 80% of the way there quickly, but you still need domain knowledge and the ability to debug when their information is stale or incorrect.

Self-Review: Getting Copilot to Improve Its Own Code

One powerful technique that addresses concerns about AI-generated code quality is using Copilot to review and improve the very code it has produced. Critics might argue that AI assistants lack sufficient context while working incrementally, potentially leading to suboptimal solutions. However, this misses a crucial capability: asking the AI to review the solution as a whole.

After Copilot had implemented the core functionality, I took a step back and asked:

Me: “Review the entire codebase and identify any issues, inefficiencies, or areas that could be improved.”

This prompted a comprehensive analysis that surfaced real problems I hadn’t noticed during rapid feature development:

🔴 High Priority Issues

1. Duplicate formatBytes function - Defined in 4 separate files (MemoryMonitor.tsx, DiskMonitor.tsx, NetworkMonitor.tsx, ProcessList.tsx), each with slightly different implementations
2. Type safety issues in App.tsx - Using any types for state variables like systemInfo, processes, and batteryInfo instead of proper interfaces
3. Preload types don’t match actual API - The ProcessInfo interface was missing many fields the Rust module actually returns (ppid, runTime, cpuTime, status, command, diskRead, diskWrite, isThread)

🟡 Medium Priority Issues

1. Native module type is any - No TypeScript declarations for the Rust bindings
2. Duplicate formatStorage function - Defined in both DiskMonitor.tsx and NetworkMonitor.tsx
3. ChartJS registered multiple times - Each monitor component was independently registering ChartJS components, should be done once at app level
4. Large inline styles in App.tsx header - The battery status section (188-275 lines) had extensive inline styles that could be extracted to CSS
5. ProcessList.tsx is 664 lines - Very large component that could be split into ProcessTable.tsx, ProcessControls.tsx, and ConfirmDialog.tsx

🟢 Low Priority

1. CSS variables defined but not fully used - Variables like --spacing-xs and --font-sm were defined but many inline styles used hardcoded values
2. Missing React.memo on monitor components - Could prevent unnecessary re-renders on gauge/chart components
3. Console.error for non-critical failures - Should use proper logging utility or remove in production

For each issue, Copilot provided context and suggested fixes. The conversation flowed naturally:

Me: “Fix the duplicate formatBytes functions”

Copilot: [Creates src/utils/formatters.ts with a single, comprehensive implementation and updates all 4 files to import it]

Me: “Add proper TypeScript types for the system info state”

Copilot: [Creates interface definitions matching the Rust module’s output and updates all any types to use them]

This self-review pattern is remarkably effective because:

1. Full context awareness - During review, the AI can see all the pieces together, not just the immediate task
2. Pattern recognition - It can spot anti-patterns across the codebase that emerge from incremental development
3. Best practices application - It can suggest idiomatic improvements that might have been missed during rapid feature development
4. Consistency checking - It identifies places where similar functionality is handled differently

The key is being explicit about the review scope. Rather than asking a vague “make it better,” specific prompts work best:

• “Review the React components for performance issues”
• “Check the Rust module for error handling problems”
• “Analyze the IPC communication for potential race conditions”
• “Review accessibility and ensure proper ARIA labels”

This approach transforms a potential weakness into a strength. Yes, AI might produce less-than-perfect code when working with limited context on individual features. But it excels at identifying those imperfections when given the full picture and asked to review holistically.

Setting Up the Release Pipeline

One area where AI assistance proved invaluable was configuring the build and release pipeline. CI/CD configuration is notoriously fiddly, with countless options and edge cases.

Me: “I would like to allow people finding this GitHub repository to be able to download and install the application. What options do I have?”

Copilot outlined several distribution strategies (GitHub Releases, Homebrew, Mac App Store) and recommended GitHub Releases with electron-builder for an open-source project. It then generated:

• A build.yml workflow for CI on every push
• A release.yml workflow triggered by version tags
• Electron-builder configuration for both Intel and Apple Silicon
• README updates with download instructions

When the workflows hit issues; wrong action names, permission errors, DMG creation failures on CI runners, each problem was solved through conversation:

• “The action dtolnay/rust-action@stable does not exist” → Switched to actions-rust-lang/setup-rust-toolchain@v1
• “Package ‘electron’ is only allowed in devDependencies” → Moved the dependency
• “hdiutil detach failing on CI” → Added retry logic for flaky DMG creation
• “403 Forbidden when creating release” → Added permissions: contents: write

Lessons Learned

When Vibe Coding Excels

1. Boilerplate and configuration - Webpack configs, GitHub Actions, package.json scripts
2. API integration - Knowing which Electron or React APIs to use
3. Cross-referencing - Using data from one part of the app in another
4. Bug diagnosis - Describing symptoms and getting targeted fixes
5. Refactoring - “Can you change this from flex to grid layout?”

When to Slow Down

1. Architecture decisions - Still worth thinking through yourself
2. Security-sensitive code - Review carefully, don’t blindly trust
3. Complex algorithms - Understand what’s generated, don’t just accept
4. Performance-critical paths - Verify the approach makes sense

The Collaboration Dynamic

The most effective pattern I found was treating Copilot as a highly knowledgeable pair programmer who happens to type very fast. I would:

1. Set context - Explain what exists and what I’m trying to achieve
2. Ask for options - “What are the ways to do X?”
3. Make decisions - Choose the approach that fits best
4. Request implementation - “Let’s go with option 2”
5. Iterate on details - “Can we also handle the edge case where…”

This maintains human agency over the architecture and design while leveraging AI for implementation speed.

The Results

Peep went from concept to released application in a remarkably short time. The final product includes:

• 5 monitoring components (CPU, Memory, Disk, Network, Processes)
• Native Rust module with 15+ exported functions
• Automated CI/CD with multi-architecture builds
• GitHub Releases with DMG and ZIP downloads
• Comprehensive documentation (README, CONTRIBUTING, CHANGELOG)

More importantly, the codebase is clean and maintainable. The AI-generated code follows consistent patterns, includes appropriate comments, and handles edge cases I might have missed.

Conclusion

Building Peep reinforced my belief that we’re at an inflection point in software development. The combination of vibe coding (describing intent in natural language) with tools like GitHub Copilot Chat doesn’t replace programming skill; it amplifies it.

You still need to understand what you’re building, make architectural decisions, and verify the results. But the mechanical translation of ideas into code? That’s increasingly handled by AI, freeing developers to focus on the creative and strategic aspects of software creation.

If you haven’t tried this approach, I encourage you to experiment. Start with a side project, describe what you want, and see where the conversation takes you. You might be surprised how quickly your ideas become reality.

Peep is open source and available on GitHub. Contributions welcome!

Docker Bake solves these challenges with a declarative approach to building container images. It’s a high-level build orchestration tool that lets you define complex build configurations in a single file, then execute them consistently across local development and CI/CD pipelines. Instead of maintaining dozens of docker build commands with slightly different flags, you define your entire build matrix once and let Bake handle the orchestration.

In this comprehensive guide, I’ll show you how to use Docker Bake to build production-ready container images. We’ll start with the fundamentals, progress through real-world examples of multi-platform and multi-tag builds, and finish with GitHub Actions workflows that demonstrate how to integrate Bake into your CI/CD pipeline. By the end, you’ll have the knowledge to replace your complex build scripts with maintainable, declarative configuration.

Understanding Docker Bake

Before diving into examples, it’s worth understanding what Docker Bake actually does and why it exists. Docker Bake is part of Docker Buildx, the extended build capabilities that replaced the classic Docker builder. Whilst docker buildx build handles building a single image, docker bake orchestrates building multiple related images in one operation.

Why Docker Bake Exists

Traditional Docker builds work well for simple use cases; a single Dockerfile producing a single image. But real world applications often need more complexity. You might need to build the same application for both AMD64 and ARM64 platforms. You might maintain separate images with different base operating systems. You might need to tag the same build with multiple version identifiers.

Without Bake, you’d write shell scripts that loop through platforms, base images, and tags, invoking docker build repeatedly with different arguments. These scripts become difficult to maintain, hard to test, and prone to subtle bugs where one combination of flags differs slightly from another.

Bake replaces these scripts with declarative configuration. You define what you want to build, not how to build it. The configuration format supports variables, inheritance, and composition, making it easier to maintain consistency across dozens of build targets.

Bake Configuration Format

Docker Bake supports three configuration formats: HCL (HashiCorp Configuration Language), JSON, and Docker Compose files. HCL is the most expressive and the format I recommend. It supports variables, functions, and a clean syntax that reads naturally.

A Bake file consists of targets, groups, and variables. Targets define individual build operations; which Dockerfile to use, what tags to apply, which platforms to build for. Groups collect multiple targets so you can build them together. Variables parameterise your configuration, allowing the same file to work across different environments.

How Bake Differs from Regular Builds

When you run docker buildx build, you’re building a single image. When you run docker buildx bake, you can build dozens of images in parallel, each with different configurations, all defined in your Bake file. Bake also provides better defaults for multi-platform builds, automatically configuring builders and handling the complexity of cross-compilation.

Another key difference is composability. Bake files can reference other Bake files, allowing you to split configuration across multiple files and compose them at build time. This makes it possible to maintain shared configuration whilst allowing project-specific overrides.

Basic Docker Bake Configuration

Let’s start with a simple example and build up complexity progressively. Create a file called docker-bake.hcl:

variable "TAG" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

group "default" {
  targets = ["app"]
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  tags       = ["${REGISTRY}/myapp:${TAG}"]
  platforms  = ["linux/amd64"]
}

This configuration defines a single target called app that builds your Dockerfile and tags it with a registry prefix and version tag. The default group contains this target, so running docker buildx bake without arguments builds it.

Variables make this configuration flexible. Override them at runtime without modifying the file:

docker buildx bake --set "*.TAG=1.0.0"

The * pattern applies the variable to all targets. You can also set variables through environment variables or by creating a docker-bake.override.hcl file that gets automatically merged with the main configuration.

Adding Context and Arguments

Real applications need build arguments for things like version numbers, build timestamps, or configuration values. Add them to your target:

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  tags       = ["${REGISTRY}/myapp:${TAG}"]
  platforms  = ["linux/amd64"]
  
  args = {
    VERSION    = "${TAG}"
    BUILD_DATE = timestamp()
    VCS_REF    = "git-sha"
  }
  
  labels = {
    "org.opencontainers.image.version"    = "${TAG}"
    "org.opencontainers.image.created"    = timestamp()
    "org.opencontainers.image.source"     = "https://github.com/myorg/myapp"
  }
}

The args block passes build arguments to your Dockerfile, accessible via ARG instructions. The labels block adds OCI-compliant metadata to your image. The timestamp() function generates the current timestamp, ensuring each build has accurate creation metadata.

Multi-Platform Builds

Multi-platform builds are where Docker Bake really shines. Building for multiple architectures with traditional Docker builds requires multiple commands, managing different builders, and complex scripting. Bake handles all of this automatically.

Configuring Platform Support

Update your target to build for both AMD64 and ARM64:

variable "TAG" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  tags       = ["${REGISTRY}/myapp:${TAG}"]
  platforms  = PLATFORMS
  
  args = {
    BUILDPLATFORM = ""
    TARGETPLATFORM = ""
  }
}

When you run docker buildx bake, Bake builds your image for both platforms in parallel, creating a multi-architecture manifest that automatically serves the correct image based on the host architecture.

The BUILDPLATFORM and TARGETPLATFORM arguments are automatically available in your Dockerfile when building multi-platform images. They let you customise build behaviour based on the target architecture:

FROM --platform=$BUILDPLATFORM golang:1.21 AS builder

ARG TARGETPLATFORM
ARG BUILDPLATFORM

RUN echo "Building on $BUILDPLATFORM for $TARGETPLATFORM"

WORKDIR /app
COPY . .

# Use TARGETPLATFORM to set GOARCH appropriately
RUN case "$TARGETPLATFORM" in \
      "linux/amd64") GOARCH=amd64 ;; \
      "linux/arm64") GOARCH=arm64 ;; \
      "linux/arm/v7") GOARCH=arm GOARM=7 ;; \
    esac && \
    GOOS=linux go build -o myapp

FROM alpine:3.19
COPY --from=builder /app/myapp /usr/local/bin/
ENTRYPOINT ["/usr/local/bin/myapp"]

Platform-Specific Optimisations

Sometimes you need platform-specific behaviour. For ARM builds, you might want to use a different base image or enable specific compiler optimisations. Create separate targets for each platform:

variable "TAG" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

target "app-amd64" {
  inherits   = ["app-common"]
  platforms  = ["linux/amd64"]
  tags       = ["${REGISTRY}/myapp:${TAG}-amd64"]
  
  args = {
    BASE_IMAGE = "alpine:3.19"
  }
}

target "app-arm64" {
  inherits   = ["app-common"]
  platforms  = ["linux/arm64"]
  tags       = ["${REGISTRY}/myapp:${TAG}-arm64"]
  
  args = {
    BASE_IMAGE = "arm64v8/alpine:3.19"
  }
}

target "app-common" {
  context    = "."
  dockerfile = "Dockerfile"
  
  args = {
    VERSION = "${TAG}"
  }
  
  labels = {
    "org.opencontainers.image.version" = "${TAG}"
  }
}

group "default" {
  targets = ["app-amd64", "app-arm64"]
}

This configuration uses inheritance through the inherits field. The app-common target defines shared configuration, whilst platform-specific targets override only what differs. Building the default group builds both platforms with their specific optimisations.

Multi-Tag Strategies

Production container workflows often need multiple tags pointing to the same image. You might tag a release as 1.2.3, 1.2, 1, and latest simultaneously. You might also tag builds with git commit SHAs for traceability. Bake makes this straightforward.

Semantic Versioning Tags

Here’s a configuration that tags images with multiple semantic version levels:

variable "VERSION" {
  default = "1.2.3"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "COMMIT_SHA" {
  default = "unknown"
}

function "semver_tags" {
  params = [version]
  result = [
    "${REGISTRY}/myapp:${version}",
    "${REGISTRY}/myapp:${regex("^[0-9]+\\.[0-9]+", version)}",
    "${REGISTRY}/myapp:${regex("^[0-9]+", version)}",
    "${REGISTRY}/myapp:latest"
  ]
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = semver_tags(VERSION)
  
  args = {
    VERSION    = VERSION
    COMMIT_SHA = COMMIT_SHA
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.revision" = COMMIT_SHA
  }
}

The semver_tags function takes a version like 1.2.3 and generates four tags: 1.2.3, 1.2, 1, and latest. This ensures users can pin to specific versions or track major/minor releases.

Environment-Specific Tags

Different environments often need different tagging strategies. Development builds might use branch names and commit SHAs, whilst production uses semantic versions:

variable "ENVIRONMENT" {
  default = "dev"
}

variable "VERSION" {
  default = "dev"
}

variable "BRANCH" {
  default = "main"
}

variable "COMMIT_SHA" {
  default = "unknown"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

function "dev_tags" {
  params = [branch, sha]
  result = [
    "${REGISTRY}/myapp:${branch}",
    "${REGISTRY}/myapp:${branch}-${sha}",
    "${REGISTRY}/myapp:dev"
  ]
}

function "prod_tags" {
  params = [version]
  result = [
    "${REGISTRY}/myapp:${version}",
    "${REGISTRY}/myapp:${regex("^[0-9]+\\.[0-9]+", version)}",
    "${REGISTRY}/myapp:${regex("^[0-9]+", version)}",
    "${REGISTRY}/myapp:latest"
  ]
}

target "app-dev" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64"]
  tags       = dev_tags(BRANCH, COMMIT_SHA)
}

target "app-prod" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = prod_tags(VERSION)
  
  output     = ["type=registry,push=true"]
}

group "dev" {
  targets = ["app-dev"]
}

group "prod" {
  targets = ["app-prod"]
}

Build development images with docker buildx bake dev and production images with docker buildx bake prod. Each group uses appropriate tags and platforms for its environment.

Multiple Base Images

Some applications need variants built from different base images. You might offer an Alpine-based image for minimal size and a Debian-based image for better compatibility. Or you might support multiple runtime versions—Node.js 18 and Node.js 20, for example.

Base Image Variants

Here’s a configuration that builds the same application with multiple base images:

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

target "app-alpine" {
  context    = "."
  dockerfile = "Dockerfile.alpine"
  platforms  = PLATFORMS
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-alpine",
    "${REGISTRY}/myapp:alpine"
  ]
  
  args = {
    BASE_IMAGE = "alpine:3.19"
    VERSION    = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.base.name" = "alpine:3.19"
  }
}

target "app-debian" {
  context    = "."
  dockerfile = "Dockerfile.debian"
  platforms  = PLATFORMS
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-debian",
    "${REGISTRY}/myapp:${VERSION}",
    "${REGISTRY}/myapp:latest"
  ]
  
  args = {
    BASE_IMAGE = "debian:bookworm-slim"
    VERSION    = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.base.name" = "debian:bookworm-slim"
  }
}

target "app-ubuntu" {
  context    = "."
  dockerfile = "Dockerfile.ubuntu"
  platforms  = PLATFORMS
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-ubuntu",
    "${REGISTRY}/myapp:ubuntu"
  ]
  
  args = {
    BASE_IMAGE = "ubuntu:22.04"
    VERSION    = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
    "org.opencontainers.image.base.name" = "ubuntu:22.04"
  }
}

group "default" {
  targets = ["app-alpine", "app-debian", "app-ubuntu"]
}

group "alpine" {
  targets = ["app-alpine"]
}

group "debian" {
  targets = ["app-debian"]
}

Running docker buildx bake builds all three variants. Running docker buildx bake alpine builds only the Alpine variant. Each variant has appropriate tags indicating its base image.

Single Dockerfile with Multiple Bases

If your Dockerfiles differ only in the base image, you can use a single Dockerfile with build arguments:

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

target "app-alpine" {
  inherits   = ["app-common"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-alpine"]
  
  args = {
    BASE_IMAGE = "alpine:3.19"
    IMAGE_VARIANT = "alpine"
  }
}

target "app-debian" {
  inherits   = ["app-common"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-debian", "${REGISTRY}/myapp:${VERSION}"]
  
  args = {
    BASE_IMAGE = "debian:bookworm-slim"
    IMAGE_VARIANT = "debian"
  }
}

target "app-common" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  
  args = {
    VERSION = VERSION
  }
}

group "default" {
  targets = ["app-alpine", "app-debian"]
}

Your Dockerfile uses the BASE_IMAGE argument:

ARG BASE_IMAGE=alpine:3.19
FROM ${BASE_IMAGE}

ARG VERSION
ARG IMAGE_VARIANT

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.variant="${IMAGE_VARIANT}"

WORKDIR /app
COPY . .

RUN if [ "${IMAGE_VARIANT}" = "alpine" ]; then \
      apk add --no-cache ca-certificates; \
    else \
      apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*; \
    fi

ENTRYPOINT ["/app/myapp"]

This approach reduces duplication when the only difference between variants is the base image and package manager.

Runtime Version Matrix

Applications often need to support multiple runtime versions. A Python application might support Python 3.10, 3.11, and 3.12. A Node.js application might support Node 18, 20, and 21. Bake makes it easy to build a complete matrix.

Python Version Matrix Example

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

variable "PYTHON_VERSIONS" {
  default = ["3.10", "3.11", "3.12"]
}

target "app-python-310" {
  inherits   = ["app-python-common"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python3.10",
    "${REGISTRY}/myapp:python3.10"
  ]
  
  args = {
    PYTHON_VERSION = "3.10"
    PYTHON_IMAGE = "python:3.10-slim"
  }
}

target "app-python-311" {
  inherits   = ["app-python-common"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python3.11",
    "${REGISTRY}/myapp:python3.11"
  ]
  
  args = {
    PYTHON_VERSION = "3.11"
    PYTHON_IMAGE = "python:3.11-slim"
  }
}

target "app-python-312" {
  inherits   = ["app-python-common"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python3.12",
    "${REGISTRY}/myapp:${VERSION}",
    "${REGISTRY}/myapp:latest"
  ]
  
  args = {
    PYTHON_VERSION = "3.12"
    PYTHON_IMAGE = "python:3.12-slim"
  }
}

target "app-python-common" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = PLATFORMS
  
  args = {
    VERSION = VERSION
  }
  
  labels = {
    "org.opencontainers.image.version" = VERSION
  }
}

group "default" {
  targets = ["app-python-310", "app-python-311", "app-python-312"]
}

group "python-latest" {
  targets = ["app-python-312"]
}

The corresponding Dockerfile:

ARG PYTHON_IMAGE=python:3.12-slim
FROM ${PYTHON_IMAGE}

ARG VERSION
ARG PYTHON_VERSION

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.python.version="${PYTHON_VERSION}"

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["python", "app.py"]

Advanced Bake Patterns

Once you’re comfortable with basic Bake configurations, several advanced patterns can make your builds even more powerful.

Matrix Builds with Dynamic Targets

For large build matrices, manually defining every combination becomes tedious. Whilst Bake doesn’t support true matrix generation in HCL, you can use environment variables and multiple Bake files to achieve similar results.

Create a base configuration docker-bake.hcl:

variable "VERSION" {
  default = "latest"
}

variable "REGISTRY" {
  default = "docker.io/myorg"
}

variable "PYTHON_VERSION" {
  default = "3.12"
}

variable "BASE_VARIANT" {
  default = "slim"
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = [
    "${REGISTRY}/myapp:${VERSION}-python${PYTHON_VERSION}-${BASE_VARIANT}"
  ]
  
  args = {
    PYTHON_IMAGE = "python:${PYTHON_VERSION}-${BASE_VARIANT}"
    VERSION      = VERSION
  }
}

Then use a shell script to invoke Bake multiple times with different variables:

#!/bin/bash
set -e

VERSIONS=("3.10" "3.11" "3.12")
VARIANTS=("slim" "alpine")

for version in "${VERSIONS[@]}"; do
  for variant in "${VARIANTS[@]}"; do
    echo "Building Python ${version} ${variant}"
    docker buildx bake \
      --set "*.PYTHON_VERSION=${version}" \
      --set "*.BASE_VARIANT=${variant}" \
      app
  done
done

Conditional Building

Sometimes you want to build certain targets only in specific conditions. Whilst Bake doesn’t support conditionals directly, you can use groups and selective building:

variable "BUILD_ARCH" {
  default = "all"
}

target "app-amd64" {
  inherits   = ["app-common"]
  platforms  = ["linux/amd64"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-amd64"]
}

target "app-arm64" {
  inherits   = ["app-common"]
  platforms  = ["linux/arm64"]
  tags       = ["${REGISTRY}/myapp:${VERSION}-arm64"]
}

target "app-common" {
  context    = "."
  dockerfile = "Dockerfile"
}

group "all" {
  targets = ["app-amd64", "app-arm64"]
}

group "amd64" {
  targets = ["app-amd64"]
}

group "arm64" {
  targets = ["app-arm64"]
}

Build only AMD64 with docker buildx bake amd64, only ARM64 with docker buildx bake arm64, or both with docker buildx bake all.

Build Secrets and SSH Forwarding

Production builds often need access to private dependencies or repositories. Bake supports BuildKit secrets and SSH forwarding:

variable "VERSION" {
  default = "latest"
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = ["linux/amd64", "linux/arm64"]
  tags       = ["myapp:${VERSION}"]
  
  secret = [
    "id=npm_token,env=NPM_TOKEN",
    "id=github_token,env=GITHUB_TOKEN"
  ]
  
  ssh = ["default"]
}

In your Dockerfile, mount secrets:

FROM node:20-alpine

WORKDIR /app

COPY package*.json ./

# Mount npm token as secret for private registry access
RUN --mount=type=secret,id=npm_token \
    echo "//registry.npmjs.org/:_authToken=$(cat /run/secrets/npm_token)" > .npmrc && \
    npm ci && \
    rm .npmrc

COPY . .

RUN npm run build

CMD ["npm", "start"]

For SSH access to private Git repositories:

FROM node:20-alpine

RUN apk add --no-cache git openssh-client

WORKDIR /app

# Mount SSH socket for private repo access
RUN --mount=type=ssh \
    mkdir -p ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts && \
    npm install git+ssh://git@github.com/myorg/private-package.git

COPY . .

CMD ["npm", "start"]

GitHub Actions Integration

Docker Bake integrates seamlessly with GitHub Actions, providing a clean way to build and push container images in CI/CD pipelines.

Basic GitHub Actions Workflow

Create .github/workflows/docker-build.yml:

name: Build and Push Docker Images

on:
  push:
    branches:
      - main
      - develop
    tags:
      - 'v*'
  pull_request:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract metadata
        id: meta
        run: |
          VERSION=latest
          if [[ $GITHUB_REF == refs/tags/v* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          elif [[ $GITHUB_REF == refs/heads/main ]]; then
            VERSION=main
          elif [[ $GITHUB_REF == refs/heads/* ]]; then
            VERSION=${GITHUB_REF#refs/heads/}
          fi
          
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
          echo "commit_sha=${GITHUB_SHA::8}" >> $GITHUB_OUTPUT
          echo "build_date=$(date -u +'%Y-%m-%dT%H:%M:%SZ')" >> $GITHUB_OUTPUT
      
      - name: Build images
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          set: |
            *.VERSION=$
            *.COMMIT_SHA=$
            *.REGISTRY=$/$
          push: $

This workflow builds your images on every push and pull request, pushing only when changes are merged to main or when tags are created.

Multi-Platform Builds in GitHub Actions

For multi-platform builds, configure QEMU emulation:

name: Build Multi-Platform Images

on:
  push:
    tags:
      - 'v*'

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          platforms: 'arm64,arm'
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract version
        id: version
        run: |
          VERSION=${GITHUB_REF#refs/tags/v}
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
      
      - name: Build and push multi-platform images
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          targets: default
          set: |
            *.VERSION=$
            *.REGISTRY=$/$
            *.PLATFORMS=linux/amd64,linux/arm64
          push: true

Matrix Builds in GitHub Actions

For building multiple variants, use GitHub Actions matrix strategy:

name: Build Image Matrix

on:
  push:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    
    strategy:
      matrix:
        python-version: ['3.10', '3.11', '3.12']
        base-variant: ['slim', 'alpine']
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Build and push
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          targets: app
          set: |
            *.PYTHON_VERSION=$
            *.BASE_VARIANT=$
            *.REGISTRY=$/$
          push: true

This workflow builds all combinations of Python versions and base variants in parallel.

Build Caching in CI

BuildKit caching dramatically speeds up CI builds. Configure cache exports and imports:

name: Build with Caching

on:
  push:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Build and push with cache
        uses: docker/bake-action@v4
        with:
          files: |
            ./docker-bake.hcl
          set: |
            *.REGISTRY=$/$
            *.cache-from=type=registry,ref=$/$:buildcache
            *.cache-to=type=registry,ref=$/$:buildcache,mode=max
          push: true

Update your docker-bake.hcl to support cache configuration:

variable "CACHE_FROM" {
  default = ""
}

variable "CACHE_TO" {
  default = ""
}

target "app" {
  context    = "."
  dockerfile = "Dockerfile"
  
  cache-from = CACHE_FROM != "" ? [CACHE_FROM] : []
  cache-to   = CACHE_TO != "" ? [CACHE_TO] : []
}

Reusable Workflow for Multiple Repositories

Create a reusable workflow that can be used across multiple repositories. In your organisation’s .github repository, create .github/workflows/docker-bake-reusable.yml:

name: Reusable Docker Bake Workflow

on:
  workflow_call:
    inputs:
      bake-file:
        description: 'Path to docker-bake.hcl file'
        required: false
        type: string
        default: './docker-bake.hcl'
      bake-target:
        description: 'Bake target to build'
        required: false
        type: string
        default: 'default'
      registry:
        description: 'Container registry'
        required: false
        type: string
        default: 'ghcr.io'
      platforms:
        description: 'Platforms to build for'
        required: false
        type: string
        default: 'linux/amd64,linux/arm64'
      push:
        description: 'Push images to registry'
        required: false
        type: boolean
        default: true
    secrets:
      registry-username:
        description: 'Registry username'
        required: false
      registry-password:
        description: 'Registry password'
        required: true

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract metadata
        id: meta
        run: |
          VERSION=latest
          if [[ $GITHUB_REF == refs/tags/v* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          elif [[ $GITHUB_REF == refs/heads/main ]]; then
            VERSION=main
          fi
          
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
          echo "commit_sha=${GITHUB_SHA::8}" >> $GITHUB_OUTPUT
      
      - name: Build and push
        uses: docker/bake-action@v4
        with:
          files: $
          targets: $
          set: |
            *.VERSION=$
            *.COMMIT_SHA=$
            *.REGISTRY=$/$
            *.PLATFORMS=$
          push: $

Use it in your application repositories:

name: Build Docker Images

on:
  push:
    branches:
      - main
    tags:
      - 'v*'

jobs:
  build:
    uses: myorg/.github/.github/workflows/docker-bake-reusable.yml@main
    with:
      platforms: 'linux/amd64,linux/arm64'
      push: true
    secrets:
      registry-password: $

Complete Real-World Example

Let’s put everything together with a complete example: a Node.js application that supports multiple Node versions, multiple platforms, and builds both development and production variants.

Project Structure

.
├── docker-bake.hcl
├── docker-bake.override.hcl.example
├── Dockerfile
├── package.json
├── src/
│   └── app.js
└── .github/
    └── workflows/
        └── docker.yml

docker-bake.hcl

variable "VERSION" {
  default = "dev"
}

variable "REGISTRY" {
  default = "ghcr.io/myorg"
}

variable "COMMIT_SHA" {
  default = "unknown"
}

variable "BUILD_DATE" {
  default = ""
}

variable "NODE_VERSIONS" {
  default = ["18", "20", "21"]
}

variable "PLATFORMS" {
  default = ["linux/amd64", "linux/arm64"]
}

function "semver_tags" {
  params = [registry, app, version, node_version]
  result = [
    "${registry}/${app}:${version}-node${node_version}",
    "${registry}/${app}:node${node_version}"
  ]
}

function "prod_tags" {
  params = [registry, app, version, node_version]
  result = concat(
    semver_tags(registry, app, version, node_version),
    node_version == "20" ? [
      "${registry}/${app}:${version}",
      "${registry}/${app}:latest"
    ] : []
  )
}

# Node 18 targets
target "app-node18-dev" {
  inherits   = ["node-common"]
  tags       = ["${REGISTRY}/myapp:dev-node18"]
  args = {
    NODE_VERSION = "18"
    NODE_ENV = "development"
  }
  platforms = ["linux/amd64"]
}

target "app-node18-prod" {
  inherits   = ["node-common"]
  tags       = prod_tags(REGISTRY, "myapp", VERSION, "18")
  args = {
    NODE_VERSION = "18"
    NODE_ENV = "production"
  }
  output = ["type=registry,push=true"]
}

# Node 20 targets
target "app-node20-dev" {
  inherits   = ["node-common"]
  tags       = ["${REGISTRY}/myapp:dev-node20", "${REGISTRY}/myapp:dev"]
  args = {
    NODE_VERSION = "20"
    NODE_ENV = "development"
  }
  platforms = ["linux/amd64"]
}

target "app-node20-prod" {
  inherits   = ["node-common"]
  tags       = prod_tags(REGISTRY, "myapp", VERSION, "20")
  args = {
    NODE_VERSION = "20"
    NODE_ENV = "production"
  }
  output = ["type=registry,push=true"]
}

# Node 21 targets
target "app-node21-dev" {
  inherits   = ["node-common"]
  tags       = ["${REGISTRY}/myapp:dev-node21"]
  args = {
    NODE_VERSION = "21"
    NODE_ENV = "development"
  }
  platforms = ["linux/amd64"]
}

target "app-node21-prod" {
  inherits   = ["node-common"]
  tags       = prod_tags(REGISTRY, "myapp", VERSION, "21")
  args = {
    NODE_VERSION = "21"
    NODE_ENV = "production"
  }
  output = ["type=registry,push=true"]
}

# Common target configuration
target "node-common" {
  context    = "."
  dockerfile = "Dockerfile"
  platforms  = PLATFORMS
  
  args = {
    VERSION    = VERSION
    COMMIT_SHA = COMMIT_SHA
    BUILD_DATE = BUILD_DATE != "" ? BUILD_DATE : timestamp()
  }
  
  labels = {
    "org.opencontainers.image.version"    = VERSION
    "org.opencontainers.image.revision"   = COMMIT_SHA
    "org.opencontainers.image.created"    = BUILD_DATE != "" ? BUILD_DATE : timestamp()
    "org.opencontainers.image.source"     = "https://github.com/myorg/myapp"
    "org.opencontainers.image.vendor"     = "MyOrg"
  }
  
  cache-from = ["type=registry,ref=${REGISTRY}/myapp:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/myapp:buildcache,mode=max"]
}

# Groups for different scenarios
group "dev" {
  targets = ["app-node18-dev", "app-node20-dev", "app-node21-dev"]
}

group "prod" {
  targets = ["app-node18-prod", "app-node20-prod", "app-node21-prod"]
}

group "node20" {
  targets = ["app-node20-dev", "app-node20-prod"]
}

group "default" {
  targets = ["app-node20-dev"]
}

Dockerfile

ARG NODE_VERSION=20
FROM node:${NODE_VERSION}-alpine AS base

ARG VERSION
ARG COMMIT_SHA
ARG BUILD_DATE
ARG NODE_ENV=production

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.revision="${COMMIT_SHA}"
LABEL org.opencontainers.image.created="${BUILD_DATE}"

WORKDIR /app

# Dependencies stage
FROM base AS dependencies

COPY package*.json ./

RUN if [ "$NODE_ENV" = "production" ]; then \
      npm ci --only=production; \
    else \
      npm ci; \
    fi

# Build stage (for TypeScript or other build steps)
FROM dependencies AS build

COPY . .

RUN if [ "$NODE_ENV" = "production" ]; then \
      npm run build 2>/dev/null || true; \
    fi

# Production stage
FROM base AS production

ENV NODE_ENV=production

# Copy only production dependencies
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist 2>/dev/null || true
COPY package*.json ./
COPY src ./src

# Create non-root user
RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001 && \
    chown -R nodejs:nodejs /app

USER nodejs

EXPOSE 3000

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD node -e "require('http').get('http://localhost:3000/health', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)})"

CMD ["node", "src/app.js"]

# Development stage
FROM dependencies AS development

ENV NODE_ENV=development

COPY . .

USER node

EXPOSE 3000

CMD ["npm", "run", "dev"]

GitHub Actions Workflow

name: Build and Push Docker Images

on:
  push:
    branches:
      - main
      - develop
    tags:
      - 'v*'
  pull_request:
    branches:
      - main

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  build-dev:
    runs-on: ubuntu-latest
    if: github.event_name == 'pull_request' || github.ref == 'refs/heads/develop'
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Build development images
        uses: docker/bake-action@v4
        with:
          files: ./docker-bake.hcl
          targets: dev
          set: |
            *.REGISTRY=$/$
            *.COMMIT_SHA=$
          push: $
  
  build-prod:
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/v')
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
        with:
          platforms: 'arm64'
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract version
        id: version
        run: |
          if [[ $GITHUB_REF == refs/tags/v* ]]; then
            VERSION=${GITHUB_REF#refs/tags/v}
          else
            VERSION=main-${GITHUB_SHA::8}
          fi
          echo "version=${VERSION}" >> $GITHUB_OUTPUT
      
      - name: Build and push production images
        uses: docker/bake-action@v4
        with:
          files: ./docker-bake.hcl
          targets: prod
          set: |
            *.VERSION=$
            *.COMMIT_SHA=$
            *.REGISTRY=$/$
            *.BUILD_DATE=$
          push: true

Best Practices and Tips

After working with Docker Bake across numerous projects, several patterns have emerged as particularly effective.

Keep Bake Files Declarative

Resist the temptation to add too much logic to your Bake files. They work best when they declare what you want to build, not how to build it. Complex conditionals and computations belong in CI scripts or separate tooling, not in Bake configuration.

Use Inheritance Extensively

The inherits field is one of Bake’s most powerful features. Define common configuration once in a base target, then inherit from it in specific targets. This reduces duplication and ensures consistency across related builds.

Organise Targets into Logical Groups

Groups make it easy to build related targets together. Create groups for different scenarios: dev for development builds, prod for production, ci for continuous integration. Users can then run docker buildx bake dev without needing to know which specific targets that includes.

Version Your Bake Files

Treat your docker-bake.hcl files as code. Version them in Git, review changes through pull requests, and test modifications before merging. Bake configuration changes can have significant impact on your build process.

Document Your Targets

Add comments explaining what each target does and when to use it. Future maintainers (including yourself) will appreciate understanding the purpose of app-python-311-alpine-optimised without having to reverse-engineer the configuration.

Use Override Files for Local Development

Create a docker-bake.override.hcl.example file that developers can copy to docker-bake.override.hcl for local customisation. Bake automatically merges override files, allowing developers to adjust registry paths or platforms without modifying the main configuration.

# docker-bake.override.hcl.example
variable "REGISTRY" {
  default = "localhost:5000"
}

variable "PLATFORMS" {
  default = ["linux/amd64"]  # Build only local platform for speed
}

Validate Your Bake Files

Before committing changes, validate your Bake files:

docker buildx bake --print

This shows the resolved configuration without actually building anything, catching syntax errors and misconfigurations early.

Cache Aggressively in CI

Build caching can reduce CI build times from minutes to seconds. Configure both cache-from and cache-to in your targets and ensure your CI environment pushes cache layers to a registry.

Monitor Build Times

Track how long your builds take and optimise the slow ones. Multi-stage builds with effective layer caching make the biggest difference. BuildKit’s build statistics show where time is spent.

Troubleshooting Common Issues

Even with careful configuration, you’ll occasionally encounter issues. Here are solutions to the most common problems.

Bake Can’t Find Dockerfile

Error: failed to solve: failed to read dockerfile

This usually means the context or dockerfile path is wrong. Remember that paths in Bake files are relative to the Bake file’s location, not your current directory. Use ./ explicitly:

target "app" {
  context    = "."
  dockerfile = "./Dockerfile"
}

Platforms Not Building

If multi-platform builds fail, ensure you have the correct builder:

docker buildx create --name multiplatform --driver docker-container --use
docker buildx inspect --bootstrap

Then verify your builder supports the platforms you need:

docker buildx inspect multiplatform

Cache Not Working

If builds aren’t using cache despite configuration, check that:

1. Cache references are accessible (correct registry, authentication)
2. You’re using mode=max for cache-to to cache all layers
3. Your Dockerfile uses layer caching effectively (copy dependencies before code)

Variables Not Resolving

If variables show as literal strings instead of values, check that:

1. You’re using ${} syntax, not $ alone: ${VERSION}, not $VERSION
2. Variables are defined before use
3. You’re not mixing environment variables with Bake variables (prefix env vars with $)

Build Fails in CI But Works Locally

This often indicates environmental differences:

1. Check platform architecture (CI might be different from local)
2. Verify secrets and SSH keys are available in CI
3. Ensure network access for downloading dependencies
4. Check resource limits (CI runners might have less memory/CPU)

Conclusion

Docker Bake transforms container image builds from imperative scripts into declarative configuration. Instead of maintaining complex shell scripts that orchestrate dozens of docker build commands, you define your complete build matrix in a single file that serves as documentation, automation, and contract all at once.

The patterns we’ve explored—multi-platform builds, multiple base images, runtime version matrices, sophisticated tagging strategies—all become manageable through Bake’s inheritance, functions, and groups. Your build configuration becomes more maintainable, easier to understand, and more reliable.

For production container workflows, Bake isn’t just a convenience, it’s a force multiplier that lets you maintain complex build matrices without drowning in complexity. The GitHub Actions integrations show how Bake fits naturally into modern CI/CD pipelines, providing consistent builds from local development through production deployment.

As your containerisation needs grow, Docker Bake grows with you. Start simple with basic targets and groups, then progressively add platforms, variants, and optimisations as requirements evolve. The declarative configuration ensures that complexity remains manageable no matter how large your build matrix becomes.

The investment in learning Docker Bake pays dividends quickly. Your builds become faster through better caching, more reliable through consistent configuration, and more maintainable through clear declarative structure. You’ve built a foundation that will scale from a handful of images to hundreds of build variants without requiring fundamental rewrites.

The reality is that containers share the host kernel with every other container on the same machine. A vulnerability in that kernel affects every container. A container escape gives an attacker access to the entire host and potentially every other workload running there. Understanding these risks—and the mechanisms Linux provides to mitigate them—is essential for anyone running containers in production.

In this comprehensive guide, I’ll walk you through the fundamental security concepts that underpin container isolation, the practical measures you can implement to harden your containerised applications, and the tools and workflows that make security a natural part of your development process rather than an afterthought bolted on at deployment time.

Understanding Container Isolation

Before we can secure containers effectively, we need to understand how they achieve isolation in the first place. Containers aren’t virtual machines—they don’t run separate kernels or emulate hardware. Instead, they use Linux kernel features to create isolated environments that share the same kernel whilst appearing to be separate systems.

The Shared Kernel Model

Every container on a host shares the same Linux kernel. When you run docker run ubuntu:22.04, you’re not running an Ubuntu kernel—you’re running the host’s kernel with an Ubuntu userspace. This is fundamentally different from virtual machines, where each VM runs its own kernel instance.

This shared kernel model brings both efficiency and risk. Containers start in milliseconds because there’s no kernel to boot. They use less memory because there’s no kernel overhead per container. But a kernel vulnerability affects every container on the host, and a container escape potentially compromises every workload.

Understanding this model shapes how we think about container security. We’re not securing isolated systems; we’re securing processes that share resources with other processes whilst maintaining the illusion of isolation.

Linux Namespaces

Namespaces are the primary isolation mechanism for containers. They partition kernel resources so that one set of processes sees one set of resources whilst another set of processes sees a different set. Linux provides several namespace types, each isolating a different aspect of the system.

The PID namespace isolates process IDs. Inside a container, processes see themselves as PID 1, 2, 3, and so on, but from the host’s perspective, these same processes have entirely different PIDs. This prevents containers from seeing or signalling processes in other containers.

The network namespace isolates network interfaces, routing tables, and firewall rules. Each container gets its own network stack, appearing to have dedicated network interfaces even though the physical hardware is shared.

The mount namespace isolates filesystem mount points. A container sees its own root filesystem without access to the host’s filesystem or other containers’ filesystems.

The UTS namespace isolates hostname and domain name. Each container can have its own hostname without affecting others.

The IPC namespace isolates inter-process communication resources like shared memory segments and message queues. This prevents containers from communicating through these mechanisms unless explicitly allowed.

The user namespace maps user and group IDs inside the container to different IDs outside. Root inside the container can be mapped to an unprivileged user on the host, significantly reducing the impact of container escapes.

Here’s how you can inspect namespaces for a running container:

# Get the container's PID on the host
CONTAINER_PID=$(docker inspect --format '' my-container)

# List the namespaces for this process
ls -la /proc/$CONTAINER_PID/ns/

# Compare with host namespaces
ls -la /proc/1/ns/

Control Groups (cgroups)

While namespaces provide isolation of what a container can see, cgroups control what resources a container can use. They limit and account for CPU, memory, disk I/O, and network bandwidth consumption.

Without cgroups, a single container could consume all available memory, causing the kernel’s out-of-memory killer to terminate other containers or even critical host processes. A container could monopolise CPU, starving other workloads. Cgroups prevent this resource starvation.

# View cgroup limits for a container
docker inspect --format '' my-container
docker inspect --format '' my-container

# Set resource limits when running a container
docker run -d \
  --memory="512m" \
  --memory-swap="512m" \
  --cpus="1.5" \
  --pids-limit=100 \
  myapp:latest

The --memory flag limits total memory. The --memory-swap flag set equal to --memory disables swap, preventing the container from using swap space when memory is exhausted. The --cpus flag limits CPU usage to 1.5 cores. The --pids-limit flag prevents fork bombs by limiting the number of processes.

Linux Capabilities

Traditional Unix security divides the world into two categories: root, which can do anything, and everyone else, which is heavily restricted. This is too coarse-grained for containers. You might need a container that can bind to privileged ports but can’t load kernel modules.

Linux capabilities split root’s powers into distinct units that can be granted independently. Instead of running as root with all powers, you can run as root with only the specific capabilities needed.

# Run a container with minimal capabilities
docker run -d \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  myapp:latest

# List capabilities of a running container
docker inspect --format '' my-container
docker inspect --format '' my-container

Key capabilities to understand:

• CAP_NET_BIND_SERVICE: Bind to ports below 1024
• CAP_NET_RAW: Use raw sockets (needed for ping)
• CAP_SYS_ADMIN: A catch-all for various admin operations (avoid granting this)
• CAP_SYS_PTRACE: Trace processes (needed for debugging)
• CAP_CHOWN: Change file ownership
• CAP_SETUID / CAP_SETGID: Change user/group IDs

The principle of least privilege demands dropping all capabilities and adding back only those specifically required. Most applications need far fewer capabilities than Docker grants by default.

Seccomp Profiles

Seccomp (secure computing mode) filters system calls that a process can make. The Linux kernel provides hundreds of syscalls, but most applications need only a fraction of them. Seccomp profiles whitelist or blacklist specific syscalls, reducing the kernel attack surface available to a container.

Docker applies a default seccomp profile that blocks around 44 syscalls considered dangerous, including reboot, mount, and kexec_load. You can create custom profiles for tighter restrictions:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "defaultErrnoRet": 1,
  "architectures": [
    "SCMP_ARCH_X86_64",
    "SCMP_ARCH_AARCH64"
  ],
  "syscalls": [
    {
      "names": [
        "accept",
        "accept4",
        "bind",
        "clone",
        "close",
        "connect",
        "dup",
        "dup2",
        "execve",
        "exit",
        "exit_group",
        "fcntl",
        "fstat",
        "futex",
        "getpid",
        "getsockname",
        "getsockopt",
        "listen",
        "mmap",
        "mprotect",
        "munmap",
        "open",
        "openat",
        "read",
        "recvfrom",
        "rt_sigaction",
        "rt_sigprocmask",
        "sendto",
        "setsockopt",
        "socket",
        "write"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

Apply a custom seccomp profile:

docker run -d \
  --security-opt seccomp=/path/to/profile.json \
  myapp:latest

To generate a seccomp profile for your application, you can use tools like strace to observe which syscalls your application actually makes, then create a whitelist profile.

Image Security

Container images are the foundation of your containerised applications. A vulnerable or malicious base image compromises everything built upon it. Securing your images starts with choosing the right base and extends through scanning, signing, and supply chain verification.

Choosing Secure Base Images

The base image you choose dramatically affects your security posture. A full Ubuntu image contains thousands of packages, each potentially harbouring vulnerabilities. A minimal image contains only what’s necessary, reducing attack surface significantly.

Distroless Images

Google’s distroless images contain only your application and its runtime dependencies—no shell, no package manager, no unnecessary utilities. An attacker who gains code execution can’t spawn a shell because there isn’t one.

# Build stage
FROM golang:1.21 AS builder

WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 go build -o myapp

# Runtime stage - distroless
FROM gcr.io/distroless/static-debian12

COPY --from=builder /app/myapp /
USER nonroot:nonroot

ENTRYPOINT ["/myapp"]

Alpine Linux

Alpine uses musl libc instead of glibc and provides a minimal base around 5MB. It includes a package manager but far fewer default packages than traditional distributions.

FROM alpine:3.19

RUN apk add --no-cache ca-certificates tzdata

COPY myapp /usr/local/bin/
USER nobody:nobody

ENTRYPOINT ["/usr/local/bin/myapp"]

Scratch Images

The scratch image is literally empty; no files, no filesystem, nothing. It’s suitable for statically compiled binaries that include everything they need.

FROM golang:1.21 AS builder

WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o myapp

FROM scratch

COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=builder /app/myapp /

ENTRYPOINT ["/myapp"]

Vulnerability Scanning

Vulnerability scanners analyse your images against databases of known vulnerabilities (CVEs). They identify vulnerable packages and provide severity ratings to help prioritise remediation.

Trivy

Trivy is a comprehensive, fast scanner that checks for OS package vulnerabilities, language-specific dependencies, misconfigurations, and secrets.

# Scan an image
trivy image myapp:latest

# Scan with specific severity threshold
trivy image --severity HIGH,CRITICAL myapp:latest

# Scan and fail if vulnerabilities found (useful for CI)
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a Dockerfile for misconfigurations
trivy config Dockerfile

# Generate SBOM while scanning
trivy image --format spdx-json --output sbom.spdx.json myapp:latest

Grype

Grype from Anchore focuses specifically on vulnerability scanning with excellent accuracy and speed.

# Scan an image
grype myapp:latest

# Output as JSON for processing
grype -o json myapp:latest > vulnerabilities.json

# Fail on specific severity
grype myapp:latest --fail-on critical

Snyk

Snyk provides vulnerability scanning with remediation advice and integrates well with development workflows.

# Scan an image
snyk container test myapp:latest

# Monitor for new vulnerabilities
snyk container monitor myapp:latest

# Scan and provide fix recommendations
snyk container test myapp:latest --file=Dockerfile

Multi-Stage Builds for Security

Multi-stage builds are a security feature as much as an optimisation. They ensure build tools, source code, test data, and secrets used during build don’t end up in your final image.

# Stage 1: Dependencies
FROM node:20-alpine AS dependencies

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

# Stage 2: Build
FROM node:20-alpine AS builder

WORKDIR /app
COPY package*.json ./
RUN npm ci

COPY . .
RUN npm run build
RUN npm run test

# Stage 3: Production - only runtime artifacts
FROM node:20-alpine AS production

RUN addgroup -g 1001 -S nodejs && \
    adduser -S nodejs -u 1001

WORKDIR /app

# Copy only production dependencies
COPY --from=dependencies --chown=nodejs:nodejs /app/node_modules ./node_modules

# Copy only built artifacts
COPY --from=builder --chown=nodejs:nodejs /app/dist ./dist
COPY --from=builder --chown=nodejs:nodejs /app/package.json ./

USER nodejs

EXPOSE 3000

CMD ["node", "dist/server.js"]

Notice what doesn’t appear in the final image: npm (we use node directly), the full node_modules with dev dependencies, source TypeScript files, test files, and any secrets used during the build process. The final image contains only what’s needed to run the application.

Image Signing and Verification

Image signing cryptographically proves that an image came from a trusted source and hasn’t been tampered with. Sigstore’s Cosign has become the de facto standard for container image signing.

# Generate a key pair (for traditional key-based signing)
cosign generate-key-pair

# Sign an image with a key
cosign sign --key cosign.key myregistry/myapp:v1.0.0

# Sign using keyless signing (recommended)
# This uses OIDC identity from your CI provider
cosign sign myregistry/myapp:v1.0.0

# Verify a signature
cosign verify --key cosign.pub myregistry/myapp:v1.0.0

# Verify keyless signature
cosign verify \
  --certificate-identity=https://github.com/myorg/myapp/.github/workflows/build.yml@refs/tags/v1.0.0 \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  myregistry/myapp:v1.0.0

Keyless signing with Sigstore is particularly powerful for CI/CD. It uses your CI provider’s OIDC identity to sign images, eliminating the need to manage signing keys. The signature includes claims about who built the image and in what context, providing strong provenance guarantees.

Software Bill of Materials (SBOM)

An SBOM lists all components in your software, including libraries, dependencies, and their versions. When a new vulnerability is discovered, an SBOM lets you quickly determine which of your images are affected.

# Generate SBOM with Syft
syft myapp:latest -o spdx-json > sbom.spdx.json

# Generate SBOM with Trivy
trivy image --format spdx-json --output sbom.json myapp:latest

# Attach SBOM to image with Cosign
cosign attach sbom --sbom sbom.spdx.json myregistry/myapp:v1.0.0

# Scan SBOM for vulnerabilities
grype sbom:sbom.spdx.json

Store SBOMs alongside your images and keep them for the lifetime of the image. When Log4Shell or the next major vulnerability hits, you’ll be able to query your SBOMs to identify affected images within minutes rather than days.

Runtime Security

Securing images is essential but insufficient. Runtime security focuses on protecting containers while they execute, limiting what they can do even if compromised.

Running as Non-Root

By default, containers run as root. A vulnerability that allows code execution gives the attacker root privileges inside the container. Whilst namespaces and capabilities limit what root can do, defence in depth demands running as an unprivileged user.

FROM node:20-alpine

# Create a non-root user
RUN addgroup -g 1001 -S appgroup && \
    adduser -S appuser -u 1001 -G appgroup

WORKDIR /app

# Change ownership of application files
COPY --chown=appuser:appgroup . .

RUN npm ci --only=production

# Switch to non-root user
USER appuser

EXPOSE 3000

CMD ["node", "server.js"]

Some images require specific user IDs to match existing filesystem permissions or to integrate with orchestration platforms. Kubernetes, for example, can enforce that containers run as non-root:

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1001
    runAsGroup: 1001
    fsGroup: 1001
  containers:
    - name: app
      image: myapp:latest
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        capabilities:
          drop:
            - ALL

Read-Only Root Filesystem

A read-only root filesystem prevents attackers from modifying binaries, dropping malware, or tampering with configuration. Any write attempts fail, limiting post-exploitation options.

# Run with read-only filesystem
docker run -d \
  --read-only \
  --tmpfs /tmp:rw,noexec,nosuid,size=64m \
  --tmpfs /var/run:rw,noexec,nosuid,size=16m \
  myapp:latest

The --tmpfs mounts provide writable temporary directories in memory for applications that need to write temporary files or PID files. The noexec option prevents execution of files in these directories.

Your Dockerfile should prepare for read-only operation:

FROM node:20-alpine

RUN adduser -S appuser

WORKDIR /app
COPY --chown=appuser . .
RUN npm ci --only=production

# Ensure no writes needed to root filesystem
# Pre-create any directories the app expects to exist
RUN mkdir -p /app/cache && chown appuser /app/cache

USER appuser

CMD ["node", "server.js"]

Resource Limits

Resource limits prevent denial-of-service attacks where a compromised container exhausts host resources. They also protect against accidental resource exhaustion from memory leaks or runaway processes.

docker run -d \
  --memory="256m" \
  --memory-swap="256m" \
  --memory-reservation="128m" \
  --cpus="0.5" \
  --cpu-shares=512 \
  --pids-limit=50 \
  --ulimit nofile=1024:1024 \
  --ulimit nproc=64:64 \
  myapp:latest

In Kubernetes, specify resource requests and limits:

apiVersion: v1
kind: Pod
metadata:
  name: resource-limited-pod
spec:
  containers:
    - name: app
      image: myapp:latest
      resources:
        requests:
          memory: "128Mi"
          cpu: "250m"
        limits:
          memory: "256Mi"
          cpu: "500m"

Network Policies

Network policies control which containers can communicate with each other and with external services. By default, all containers can communicate with all other containers—a compromised container can probe and attack anything on the network.

In Kubernetes, NetworkPolicy resources define allowed traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-network-policy
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
        - namespaceSelector:
            matchLabels:
              name: monitoring
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432
    - to:
        - namespaceSelector: {}
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - protocol: UDP
          port: 53

This policy allows the API pod to receive traffic only from the frontend application and monitoring namespace, and to send traffic only to the database and DNS.

For Docker without Kubernetes, use Docker networks to isolate containers:

# Create isolated networks
docker network create --internal backend-net
docker network create frontend-net

# Database only on internal network (no external access)
docker run -d --network backend-net --name db postgres:15

# API on both networks
docker run -d --network backend-net --network frontend-net --name api myapi:latest

# Frontend only on frontend network
docker run -d --network frontend-net -p 80:80 --name web nginx:alpine

Runtime Threat Detection

Runtime security tools monitor container behaviour and alert on or block suspicious activity. They detect attacks that static scanning can’t catch—exploits against zero-day vulnerabilities, malicious insider activity, or attacks using legitimate tools.

Falco

Falco is an open-source runtime security tool that monitors syscalls and alerts on suspicious behaviour.

# Example Falco rules
- rule: Terminal Shell in Container
  desc: Detect shell spawned in a container
  condition: >
    spawned_process and 
    container and 
    shell_procs and 
    proc.tty != 0
  output: >
    Shell spawned in container 
    (user=%user.name container=%container.name shell=%proc.name 
    parent=%proc.pname cmdline=%proc.cmdline)
  priority: WARNING
  tags: [container, shell, mitre_execution]

- rule: Write Below Binary Dir
  desc: Detect writes to binary directories
  condition: >
    open_write and 
    container and 
    bin_dir
  output: >
    Write to binary directory 
    (user=%user.name container=%container.name file=%fd.name)
  priority: CRITICAL
  tags: [container, filesystem, mitre_persistence]

- rule: Outbound Connection to Unusual Port
  desc: Detect outbound connections to non-standard ports
  condition: >
    outbound and 
    container and 
    not (fd.sport in (80, 443, 53, 8080, 8443, 5432, 6379, 3306))
  output: >
    Unusual outbound connection 
    (container=%container.name connection=%fd.name)
  priority: NOTICE
  tags: [container, network, mitre_exfiltration]

Deploy Falco in Kubernetes:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app: falco
  template:
    metadata:
      labels:
        app: falco
    spec:
      serviceAccountName: falco
      hostNetwork: true
      hostPID: true
      containers:
        - name: falco
          image: falcosecurity/falco:latest
          securityContext:
            privileged: true
          volumeMounts:
            - name: dev
              mountPath: /host/dev
            - name: proc
              mountPath: /host/proc
              readOnly: true
            - name: etc
              mountPath: /host/etc
              readOnly: true
      volumes:
        - name: dev
          hostPath:
            path: /dev
        - name: proc
          hostPath:
            path: /proc
        - name: etc
          hostPath:
            path: /etc

Secret Management

Secrets in containers require careful handling. Environment variables, while convenient, can leak through logs, process listings, or debug endpoints. Build-time secrets must never persist in image layers. Runtime secrets should be injected securely and rotated regularly.

Build-Time Secrets

Docker BuildKit provides secure secret mounting during builds. Secrets are available only during the build step that mounts them and don’t persist in image layers.

# syntax=docker/dockerfile:1.4
FROM node:20-alpine

WORKDIR /app
COPY package*.json ./

# Mount npm token as secret - only available during this RUN
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc && \
    npm ci && \
    rm .npmrc

COPY . .
RUN npm run build

CMD ["node", "dist/server.js"]

Build with secrets:

# Pass secret from environment variable
export NPM_TOKEN="your-token-here"
docker buildx build --secret id=npm_token,env=NPM_TOKEN -t myapp:latest .

# Pass secret from file
echo "your-token-here" > npm_token.txt
docker buildx build --secret id=npm_token,src=npm_token.txt -t myapp:latest .

For SSH access to private repositories:

# syntax=docker/dockerfile:1.4
FROM golang:1.21 AS builder

WORKDIR /app

# Mount SSH agent for private repo access
RUN --mount=type=ssh \
    mkdir -p ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts && \
    go mod download

COPY . .
RUN go build -o myapp

FROM gcr.io/distroless/static
COPY --from=builder /app/myapp /
CMD ["/myapp"]

Build with SSH forwarding:

docker buildx build --ssh default -t myapp:latest .

Runtime Secrets

For runtime secrets, avoid environment variables where possible. Use mounted files or integrate with secret management systems.

Docker Secrets (Swarm)

# Create a secret
echo "supersecretpassword" | docker secret create db_password -

# Use in service
docker service create \
  --name api \
  --secret db_password \
  myapp:latest

Inside the container, the secret appears at /run/secrets/db_password.

Kubernetes Secrets

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  database-password: "supersecretpassword"
  api-key: "sk-1234567890"
---
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: /etc/secrets
          readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: app-secrets

External Secret Stores

For production, integrate with external secret stores like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault:

# Using External Secrets Operator with AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets-manager
  target:
    name: app-secrets
  data:
    - secretKey: database-password
      remoteRef:
        key: production/database
        property: password
    - secretKey: api-key
      remoteRef:
        key: production/api
        property: key

CI/CD Security Integration

Security must be automated into your CI/CD pipeline. Manual security reviews don’t scale, and security gates that slow down deployment get bypassed. Build security into the pipeline so it happens automatically on every build.

Comprehensive Security Pipeline

Here’s a GitHub Actions workflow that integrates vulnerability scanning, SBOM generation, and image signing:

name: Secure Container Build

on:
  push:
    branches: [main]
    tags: ['v*']
  pull_request:
    branches: [main]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: $

jobs:
  security-scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Run Trivy vulnerability scanner (filesystem)
        uses: aquasecurity/trivy-action@master
        with:
          scan-type: 'fs'
          scan-ref: '.'
          format: 'sarif'
          output: 'trivy-fs-results.sarif'
      
      - name: Upload Trivy scan results to GitHub Security
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-fs-results.sarif'

  build-and-push:
    needs: security-scan
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write  # For keyless signing
      security-events: write
    
    outputs:
      digest: $
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4
      
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      
      - name: Log in to Container Registry
        if: github.event_name != 'pull_request'
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: $/$
          tags: |
            type=ref,event=branch
            type=ref,event=pr
            type=semver,pattern=
            type=semver,pattern=.
            type=sha
      
      - name: Build and push
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: $
          tags: $
          labels: $
          cache-from: type=gha
          cache-to: type=gha,mode=max
          provenance: true
          sbom: true
      
      - name: Run Trivy vulnerability scanner (image)
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: $/$:$
          format: 'sarif'
          output: 'trivy-image-results.sarif'
          severity: 'CRITICAL,HIGH'
        if: github.event_name != 'pull_request'
      
      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v2
        with:
          sarif_file: 'trivy-image-results.sarif'
        if: github.event_name != 'pull_request'

  sign-image:
    needs: build-and-push
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    permissions:
      contents: read
      packages: write
      id-token: write
    
    steps:
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Sign the image with Cosign
        env:
          DIGEST: $
        run: |
          cosign sign --yes $/$@${DIGEST}

  generate-sbom:
    needs: build-and-push
    runs-on: ubuntu-latest
    if: github.event_name != 'pull_request'
    permissions:
      contents: read
      packages: write
    
    steps:
      - name: Install Syft
        uses: anchore/sbom-action/download-syft@v0
      
      - name: Log in to Container Registry
        uses: docker/login-action@v3
        with:
          registry: $
          username: $
          password: $
      
      - name: Generate SBOM
        env:
          DIGEST: $
        run: |
          syft $/$@${DIGEST} -o spdx-json > sbom.spdx.json
      
      - name: Upload SBOM as artifact
        uses: actions/upload-artifact@v3
        with:
          name: sbom
          path: sbom.spdx.json

Security Gates

Define security thresholds that must be met before images can be deployed:

# Add to your build job
- name: Check vulnerability threshold
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: $/$:$
    format: 'table'
    exit-code: '1'  # Fail the build
    severity: 'CRITICAL'  # Only fail on critical vulnerabilities
    ignore-unfixed: true  # Ignore vulnerabilities without fixes

- name: Verify image signature before deployment
  run: |
    cosign verify \
      --certificate-identity-regexp="https://github.com/$/*" \
      --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
      $/$@$

Admission Control

In Kubernetes, use admission controllers to enforce security policies. Kyverno and OPA Gatekeeper can verify image signatures, enforce security contexts, and block non-compliant deployments.

# Kyverno policy requiring signed images
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-images
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - entries:
                - keyless:
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/myorg/*"
---
# Policy enforcing security contexts
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-security-context
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-non-root
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Containers must run as non-root"
        pattern:
          spec:
            securityContext:
              runAsNonRoot: true
            containers:
              - securityContext:
                  allowPrivilegeEscalation: false
                  capabilities:
                    drop:
                      - ALL

Security Checklist

Here’s a comprehensive checklist for container security. Use it as a starting point and adapt it to your specific requirements:

Image Security

• Use minimal base images (distroless, Alpine, or scratch)
• Pin base image versions with digests, not just tags
• Scan images for vulnerabilities in CI/CD
• Generate and store SBOMs for all images
• Sign images with Cosign or similar
• Use multi-stage builds to exclude build tools
• Remove unnecessary packages and files
• Don’t install debugging tools in production images

Runtime Security

• Run containers as non-root users
• Drop all capabilities and add only what’s needed
• Use read-only root filesystems
• Set resource limits (memory, CPU, PIDs)
• Apply seccomp profiles
• Disable privilege escalation
• Use network policies to restrict communication
• Deploy runtime threat detection (Falco)

Secret Management

• Never commit secrets to version control
• Use BuildKit secrets for build-time credentials
• Mount secrets as files, not environment variables
• Integrate with external secret stores
• Rotate secrets regularly
• Audit secret access

CI/CD Security

• Scan code and dependencies before building
• Scan images after building
• Sign images in CI/CD pipeline
• Verify signatures before deployment
• Use admission controllers to enforce policies
• Implement security gates with severity thresholds

Infrastructure Security

• Keep container runtime updated
• Keep orchestration platform updated
• Use private registries with authentication
• Enable audit logging
• Monitor and alert on security events
• Regularly review and update security policies

Conclusion

Container security isn’t a single tool or practice—it’s a comprehensive approach that spans the entire lifecycle from base image selection through runtime monitoring. The shared kernel model means that container isolation, while effective, is not equivalent to the stronger isolation of virtual machines. Defence in depth is essential.

Start with the fundamentals: understand namespaces, cgroups, and capabilities. Choose minimal base images that reduce attack surface. Scan for vulnerabilities and generate SBOMs so you can respond quickly when new CVEs emerge. Run as non-root with minimal capabilities. Apply network policies to limit blast radius.

Automate security into your CI/CD pipeline so it happens consistently on every build. Sign your images to prove provenance. Use admission controllers to enforce policies at deployment time. Deploy runtime security tools to detect attacks that static analysis can’t catch.

Container security is an ongoing practice, not a one-time implementation. Vulnerabilities are discovered daily. Attackers evolve their techniques. Your security posture must evolve with them. Regular audits, updated policies, and continuous monitoring ensure your containerised applications remain protected as the threat landscape changes.

The investment in container security pays dividends in reduced risk, faster incident response, and confidence that your applications are protected by multiple layers of defence. You’ve built a foundation for secure containerised applications that will serve you well as your deployment grows.

For Azure-based platforms, the hub and spoke topology has emerged as the de facto standard for enterprise network design, and for good reason. It provides the perfect balance between isolation and connectivity, centralises security controls without creating bottlenecks, and scales elegantly as your organisation grows from a handful of applications to hundreds of workloads across multiple regions.

In this comprehensive guide, I’ll walk you through building a production-ready hub and spoke network topology in Azure using Terraform. We’ll explore the architectural principles that make this pattern so effective, implement the infrastructure with security best practices embedded at every layer, and address the operational considerations that separate toy examples from production-ready platforms.

Understanding the Hub and Spoke Pattern

Before we write a single line of Terraform, it’s worth understanding why this architectural pattern exists and what problems it solves. The hub and spoke topology isn’t just about drawing circles and lines on architecture diagrams, it’s a deliberate approach to managing complexity whilst maintaining security and operational efficiency.

The Problem with Flat Networks

In the early days of cloud adoption, many organisations built flat network topologies where every virtual network could communicate with every other virtual network through full-mesh peering. For three or four VNets, this works fine. By the time you reach ten VNets, you’re managing 45 peering connections. At twenty VNets, that number balloons to 190 peering connections.

Beyond the sheer management overhead, flat topologies create security challenges. Every VNet becomes a potential attack vector for every other VNet. Network security groups become impossibly complex as you try to maintain granular control over which applications can communicate. Audit trails become muddled as traffic flows directly between workloads without passing through central inspection points.

The Hub and Spoke Solution

The hub and spoke pattern solves these problems through centralisation. Instead of every VNet peering with every other VNet, spoke VNets peer only with a central hub VNet. The hub contains shared services; firewalls, VPN gateways, DNS servers, monitoring infrastructure, that all spokes can access. Traffic between spokes flows through the hub, where it can be inspected, logged, and controlled.

This topology dramatically reduces complexity. Twenty spoke VNets require only twenty peering connections to the hub, rather than 190 in a mesh topology. Security controls centralise in the hub, making them easier to maintain and audit. Network changes in one spoke don’t affect others unless you explicitly allow it.

The pattern also aligns beautifully with organisational structure. Each spoke can represent a different team, application, or environment. Development teams get their own isolated network space whilst platform engineering maintains central control over security policy, connectivity to on-premises networks, and shared infrastructure.

When to Use This Pattern

Hub and spoke topology makes sense when you have multiple workloads that need some degree of isolation from each other but also require shared services or connectivity to external networks. This describes most enterprise scenarios.

If you’re building a simple proof of concept with a single application in a single VNet, hub and spoke is overkill. But the moment you start thinking about multiple environments, multiple teams, or connections to on-premises networks, this pattern starts paying dividends. The initial investment in building the hub pays off quickly as you add spokes without increasing complexity.

Architectural Design Decisions

Every hub and spoke implementation requires making deliberate choices about how to structure the network, secure traffic, and handle connectivity. These decisions shape everything from IP address allocation to firewall rules to routing configuration.

IP Address Planning

IP address planning is one of those tasks that feels tedious until you get it wrong, at which point it becomes a nightmare. Once workloads are deployed and using IP addresses, changing them requires downtime, coordination, and significant risk.

The cardinal rule is to be generous with address space allocation. Azure supports RFC 1918 private address ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16), and there’s no cost to reserving large blocks. I recommend allocating a /16 (65,536 addresses) for the hub and at least a /20 (4,096 addresses) for each spoke, with room to grow.

Within the hub, subdivide address space by function. Allocate a subnet for Azure Firewall, another for VPN Gateway, another for Azure Bastion, and so on. Each Azure service has specific subnet sizing requirements; Azure Firewall needs at least a /26, VPN Gateway needs at least a /27, Azure Bastion needs at least a /26. Factor in future growth when sizing these subnets.

For spoke VNets, organise subnets by tier or function. Web tier gets a subnet, application tier gets another, data tier gets a third. Keep subnet sizes consistent across spokes where possible—it makes automation easier and helps with capacity planning.

Document your IP address allocation scheme from the start. Future you, six months from now, will thank present you for writing down which address ranges are allocated to which environments and teams.

Hub VNet Design

The hub VNet serves as the central point of connectivity for your entire network topology. It needs to accommodate several distinct functions, each with its own subnet requirements.

Azure Firewall sits at the heart of the hub, inspecting all traffic between spokes and controlling outbound internet access. It requires a dedicated subnet named AzureFirewallSubnet with a minimum size of /26, though /24 gives you more room for scale.

If you’re connecting to on-premises networks via VPN or ExpressRoute, you’ll need a Gateway subnet. VPN Gateway and ExpressRoute Gateway both require a subnet named GatewaySubnet with a minimum size of /27, though /26 is recommended for production deployments.

Azure Bastion provides secure RDP and SSH access to virtual machines without exposing them to the public internet. It needs a subnet named AzureBastionSubnet with a minimum size of /26.

For shared services like jump boxes, monitoring infrastructure, or centralised DNS servers, create additional subnets sized appropriately for your needs. A /27 or /28 often suffices for these management workloads.

Spoke VNet Design

Spoke VNets contain your actual workloads; the applications, databases, and services that deliver value to your organisation. Design them with isolation and security in mind.

Each spoke should map to a clear organisational or technical boundary. You might create spokes per environment (development, staging, production), per team (platform, data, mobile), per application tier (frontend, backend, data), or some combination. The key is consistency; establish a pattern and stick to it.

Within each spoke, use subnets to create security boundaries. Don’t put web servers and databases in the same subnet. Subnet boundaries allow you to apply network security groups that control which traffic is allowed. A compromised web server in its own subnet can be prevented from accessing database servers in a different subnet.

Consider creating separate subnets for private endpoints if you’re using Azure PaaS services like Storage Accounts or SQL Databases with private connectivity. This makes it easier to manage DNS resolution and network security group rules for these endpoints.

Routing Architecture

In a hub and spoke topology, routing determines how traffic flows between spokes. By default, spoke-to-spoke traffic doesn’t work even with peering configured; Azure VNet peering is non-transitive. If Spoke A peers with the Hub, and Spoke B peers with the Hub, traffic from Spoke A cannot reach Spoke B without additional configuration.

This is actually a security feature. It means you explicitly control which spokes can communicate, rather than allowing all-to-all communication by default.

To enable spoke-to-spoke communication, you have two options. The simpler approach uses Azure Firewall as a router. Configure user-defined routes (UDRs) in each spoke that send traffic destined for other spoke address ranges to the Azure Firewall. The firewall inspects the traffic and forwards it to the destination spoke.

The alternative approach deploys a network virtual appliance (NVA) in the hub for routing, but for most scenarios, Azure Firewall provides sufficient routing capabilities alongside its security functions. Using a single service for both simplifies management and reduces cost.

Connectivity Options

Your hub and spoke topology likely needs connectivity beyond Azure. On-premises datacentres, remote offices, and mobile workers all need secure access to cloud resources.

For site-to-site connectivity to on-premises networks, you’ll choose between VPN Gateway and ExpressRoute. VPN Gateway provides encrypted IPsec tunnels over the public internet and supports throughput up to 10 Gbps with the VpnGw5 SKU. It’s cost-effective and suitable for most scenarios.

ExpressRoute provides private connectivity that doesn’t traverse the public internet, with dedicated bandwidth from 50 Mbps to 100 Gbps. It costs more but provides better performance, reliability, and security for mission-critical workloads. Many organisations use both; ExpressRoute for primary connectivity and VPN Gateway as a failover path.

For remote user access, Azure Bastion provides secure RDP and SSH connectivity without exposing virtual machines to the internet. It eliminates the need for jump boxes with public IP addresses and provides audit logging of all remote access sessions.

Visual Architecture

To help visualise how all these components fit together, here’s a diagram showing a typical hub and spoke topology with three spoke VNets:

graph TB
    subgraph Internet["Internet"]
        OnPrem["On-Premises Network
192.168.0.0/16"]
        Users["Remote Users"]
    end

    subgraph Hub["Hub VNet
10.0.0.0/16"]
        subgraph FirewallSubnet["AzureFirewallSubnet
10.0.0.0/24"]
            AFW["Azure Firewall
10.0.0.4"]
        end
        
        subgraph GatewaySubnet["GatewaySubnet
10.0.1.0/24"]
            VPN["VPN Gateway"]
        end
        
        subgraph BastionSubnet["AzureBastionSubnet
10.0.2.0/24"]
            Bastion["Azure Bastion"]
        end
        
        subgraph ManagementSubnet["Management Subnet
10.0.3.0/24"]
            Monitor["Monitoring
Infrastructure"]
        end
    end

    subgraph Spoke1["Production Spoke
10.1.0.0/16"]
        subgraph WebSubnet1["Web Subnet
10.1.1.0/24"]
            Web1["Web Servers"]
        end
        subgraph AppSubnet1["App Subnet
10.1.2.0/24"]
            App1["Application
Servers"]
        end
        subgraph DataSubnet1["Data Subnet
10.1.3.0/24"]
            DB1["Databases"]
        end
        RT1["Route Table"]
    end

    subgraph Spoke2["Development Spoke
10.2.0.0/16"]
        subgraph WorkloadSubnet2["Workload Subnet
10.2.1.0/24"]
            Workload2["Development
Workloads"]
        end
        RT2["Route Table"]
    end

    subgraph Spoke3["Shared Services Spoke
10.3.0.0/16"]
        subgraph ServicesSubnet3["Services Subnet
10.3.1.0/24"]
            DNS["DNS Servers"]
            AD["Active Directory"]
        end
        RT3["Route Table"]
    end

    OnPrem -.->|"VPN Tunnel
IPsec"| VPN
    Users -.->|"RDP/SSH"| Bastion
    
    VPN -->|"Peering"| AFW
    Bastion -.->|"Secure Access"| Web1
    Bastion -.->|"Secure Access"| Workload2
    
    AFW <-->|"VNet Peering
Allow Gateway Transit"| Spoke1
    AFW <-->|"VNet Peering
Allow Gateway Transit"| Spoke2
    AFW <-->|"VNet Peering
Allow Gateway Transit"| Spoke3
    
    RT1 -.->|"0.0.0.0/0 → Firewall
10.2.0.0/16 → Firewall
10.3.0.0/16 → Firewall"| AFW
    RT2 -.->|"0.0.0.0/0 → Firewall
10.1.0.0/16 → Firewall
10.3.0.0/16 → Firewall"| AFW
    RT3 -.->|"0.0.0.0/0 → Firewall
10.1.0.0/16 → Firewall
10.2.0.0/16 → Firewall"| AFW
    
    Web1 -->|"App Traffic"| App1
    App1 -->|"Database
Queries"| DB1
    
    Workload2 -.->|"Via Firewall"| DNS
    App1 -.->|"Via Firewall"| DNS
    
    AFW -->|"Inspect & Log"| Internet
    
    Monitor -.->|"Collect Logs"| AFW
    Monitor -.->|"Collect Logs"| VPN
    Monitor -.->|"Collect Logs"| Bastion

    classDef hubStyle fill:#0078d4,stroke:#003d7a,stroke-width:2px,color:#fff
    classDef spokeStyle fill:#50e6ff,stroke:#0078d4,stroke-width:2px,color:#000
    classDef securityStyle fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff
    classDef gatewayStyle fill:#69db7c,stroke:#2b8a3e,stroke-width:2px,color:#000
    
    class AFW securityStyle
    class VPN,Bastion gatewayStyle
    class Spoke1,Spoke2,Spoke3 spokeStyle
```

Click the diagram to view fullscreen

This diagram illustrates several key architectural points. The hub sits at the centre with four distinct subnets, each serving a specific purpose. Azure Firewall acts as the central inspection point for all inter-spoke traffic and internet-bound traffic. VPN Gateway provides the encrypted tunnel to on-premises networks, whilst Azure Bastion offers secure administrative access without exposing VMs to the internet.

Each spoke VNet peers directly with the hub but not with other spokes. The route tables in each spoke contain user-defined routes that send traffic destined for other spokes through the Azure Firewall’s private IP address. This ensures all spoke-to-spoke communication flows through the firewall where it can be inspected, logged, and controlled by centralised firewall policies.

Notice how the production spoke uses a traditional three-tier subnet design (web, application, data), whilst the development spoke has a simpler structure with a single workload subnet. The shared services spoke contains infrastructure used by multiple teams, like DNS servers and Active Directory. This flexibility in spoke design is one of the pattern’s strengths—each spoke can be structured according to its specific requirements whilst maintaining consistent connectivity and security through the hub.

Infrastructure as Code with Terraform

Now let’s translate these architectural principles into actual infrastructure. We’ll build this incrementally, starting with the hub and progressively adding spokes and connectivity.

Foundation and Variables

Start by defining the variables that will drive your configuration. Create variables.tf:

variable "environment" {
  description = "Environment name (e.g., prod, staging, dev)"
  type        = string
}

variable "location" {
  description = "Primary Azure region"
  type        = string
  default     = "uksouth"
}

variable "organisation" {
  description = "Organisation name for resource naming"
  type        = string
}

variable "hub_vnet_address_space" {
  description = "Address space for hub VNet"
  type        = list(string)
  default     = ["10.0.0.0/16"]
}

variable "spoke_vnets" {
  description = "Map of spoke VNets to create"
  type = map(object({
    address_space = list(string)
    subnets = map(object({
      address_prefix = string
      service_endpoints = optional(list(string), [])
      delegation = optional(object({
        name = string
        service_delegation = object({
          name    = string
          actions = optional(list(string), [])
        })
      }))
    }))
  }))
  default = {}
}

variable "enable_vpn_gateway" {
  description = "Whether to deploy VPN Gateway in hub"
  type        = bool
  default     = true
}

variable "enable_bastion" {
  description = "Whether to deploy Azure Bastion in hub"
  type        = bool
  default     = true
}

variable "on_premises_address_spaces" {
  description = "Address spaces of on-premises networks for VPN"
  type        = list(string)
  default     = []
}

variable "tags" {
  description = "Tags to apply to all resources"
  type        = map(string)
  default     = {}
}

Create locals.tf for computed values:

locals {
  # Common tags applied to all resources
  common_tags = merge(
    var.tags,
    {
      environment = var.environment
      managed_by  = "terraform"
      pattern     = "hub-and-spoke"
    }
  )

  # Resource naming convention
  hub_name = "hub-${var.environment}-${var.location}"
  
  # Hub subnet configuration
  hub_subnets = {
    firewall = {
      name           = "AzureFirewallSubnet"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 0)
    }
    gateway = {
      name           = "GatewaySubnet"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 1)
    }
    bastion = {
      name           = "AzureBastionSubnet"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 2)
    }
    management = {
      name           = "snet-management"
      address_prefix = cidrsubnet(var.hub_vnet_address_space[0], 8, 3)
    }
  }
}

Hub Virtual Network

Create the hub VNet with all its subnets. Create hub.tf:

# Resource group for hub networking
resource "azurerm_resource_group" "hub" {
  name     = "rg-network-${local.hub_name}"
  location = var.location
  tags     = local.common_tags
}

# Hub virtual network
resource "azurerm_virtual_network" "hub" {
  name                = "vnet-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  address_space       = var.hub_vnet_address_space

  tags = local.common_tags
}

# Hub subnets
resource "azurerm_subnet" "hub_firewall" {
  name                 = local.hub_subnets.firewall.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.firewall.address_prefix]
}

resource "azurerm_subnet" "hub_gateway" {
  count = var.enable_vpn_gateway ? 1 : 0

  name                 = local.hub_subnets.gateway.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.gateway.address_prefix]
}

resource "azurerm_subnet" "hub_bastion" {
  count = var.enable_bastion ? 1 : 0

  name                 = local.hub_subnets.bastion.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.bastion.address_prefix]
}

resource "azurerm_subnet" "hub_management" {
  name                 = local.hub_subnets.management.name
  resource_group_name  = azurerm_resource_group.hub.name
  virtual_network_name = azurerm_virtual_network.hub.name
  address_prefixes     = [local.hub_subnets.management.address_prefix]

  service_endpoints = [
    "Microsoft.Storage",
    "Microsoft.KeyVault"
  ]
}

# Network security group for management subnet
resource "azurerm_network_security_group" "hub_management" {
  name                = "nsg-${local.hub_name}-management"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name

  tags = local.common_tags
}

# NSG association
resource "azurerm_subnet_network_security_group_association" "hub_management" {
  subnet_id                 = azurerm_subnet.hub_management.id
  network_security_group_id = azurerm_network_security_group.hub_management.id
}

Azure Firewall

Azure Firewall provides the central security control point for the topology. Create firewall.tf:

# Public IP for Azure Firewall
resource "azurerm_public_ip" "firewall" {
  name                = "pip-firewall-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  allocation_method   = "Static"
  sku                 = "Standard"
  zones               = ["1", "2", "3"]

  tags = local.common_tags
}

# Azure Firewall
resource "azurerm_firewall" "hub" {
  name                = "afw-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  sku_name            = "AZFW_VNet"
  sku_tier            = "Standard"
  firewall_policy_id  = azurerm_firewall_policy.hub.id
  zones               = ["1", "2", "3"]

  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.hub_firewall.id
    public_ip_address_id = azurerm_public_ip.firewall.id
  }

  tags = local.common_tags
}

# Firewall Policy
resource "azurerm_firewall_policy" "hub" {
  name                = "afwp-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  sku                 = "Standard"
  
  threat_intelligence_mode = "Alert"

  dns {
    proxy_enabled = true
  }

  intrusion_detection {
    mode = "Alert"
  }

  tags = local.common_tags
}

# Firewall Policy Rule Collection Group - Network Rules
resource "azurerm_firewall_policy_rule_collection_group" "network_rules" {
  name               = "network-rules"
  firewall_policy_id = azurerm_firewall_policy.hub.id
  priority           = 100

  network_rule_collection {
    name     = "allow-spoke-to-spoke"
    priority = 100
    action   = "Allow"

    rule {
      name                  = "allow-all-spoke-to-spoke"
      protocols             = ["Any"]
      source_addresses      = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_addresses = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_ports     = ["*"]
    }
  }

  network_rule_collection {
    name     = "allow-dns"
    priority = 110
    action   = "Allow"

    rule {
      name                  = "allow-dns-outbound"
      protocols             = ["UDP"]
      source_addresses      = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_addresses = ["*"]
      destination_ports     = ["53"]
    }
  }
}

# Firewall Policy Rule Collection Group - Application Rules
resource "azurerm_firewall_policy_rule_collection_group" "application_rules" {
  name               = "application-rules"
  firewall_policy_id = azurerm_firewall_policy.hub.id
  priority           = 200

  application_rule_collection {
    name     = "allow-azure-services"
    priority = 100
    action   = "Allow"

    rule {
      name = "allow-azure-management"
      protocols {
        type = "Https"
        port = 443
      }
      source_addresses = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_fqdns = [
        "*.azure.com",
        "*.microsoft.com",
        "*.windows.net",
        "*.azure-automation.net"
      ]
    }
  }

  application_rule_collection {
    name     = "allow-ubuntu-updates"
    priority = 110
    action   = "Allow"

    rule {
      name = "allow-apt-repositories"
      protocols {
        type = "Http"
        port = 80
      }
      protocols {
        type = "Https"
        port = 443
      }
      source_addresses = [for k, v in var.spoke_vnets : v.address_space[0]]
      destination_fqdns = [
        "*.ubuntu.com",
        "*.canonical.com"
      ]
    }
  }
}

VPN Gateway

For hybrid connectivity to on-premises networks, deploy VPN Gateway. Create vpn-gateway.tf:

# Public IP for VPN Gateway
resource "azurerm_public_ip" "vpn_gateway" {
  count = var.enable_vpn_gateway ? 1 : 0

  name                = "pip-vpngw-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  allocation_method   = "Static"
  sku                 = "Standard"
  zones               = ["1", "2", "3"]

  tags = local.common_tags
}

# VPN Gateway
resource "azurerm_virtual_network_gateway" "hub" {
  count = var.enable_vpn_gateway ? 1 : 0

  name                = "vpngw-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name

  type     = "Vpn"
  vpn_type = "RouteBased"

  active_active = false
  enable_bgp    = true
  sku           = "VpnGw2AZ"
  generation    = "Generation2"

  ip_configuration {
    name                          = "vnetGatewayConfig"
    public_ip_address_id          = azurerm_public_ip.vpn_gateway[0].id
    private_ip_address_allocation = "Dynamic"
    subnet_id                     = azurerm_subnet.hub_gateway[0].id
  }

  bgp_settings {
    asn = 65515
  }

  tags = local.common_tags
}

# Local Network Gateway for on-premises
resource "azurerm_local_network_gateway" "onpremises" {
  count = var.enable_vpn_gateway && length(var.on_premises_address_spaces) > 0 ? 1 : 0

  name                = "lng-onpremises-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name

  gateway_address = "0.0.0.0" # Replace with actual on-premises gateway IP
  address_space   = var.on_premises_address_spaces

  bgp_settings {
    asn                 = 65000
    bgp_peering_address = "192.168.1.1" # Replace with actual on-premises BGP peer IP
  }

  tags = local.common_tags
}

Azure Bastion

For secure remote access to virtual machines, deploy Azure Bastion. Create bastion.tf:

# Public IP for Azure Bastion
resource "azurerm_public_ip" "bastion" {
  count = var.enable_bastion ? 1 : 0

  name                = "pip-bastion-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  allocation_method   = "Static"
  sku                 = "Standard"
  zones               = ["1", "2", "3"]

  tags = local.common_tags
}

# Azure Bastion
resource "azurerm_bastion_host" "hub" {
  count = var.enable_bastion ? 1 : 0

  name                = "bastion-${local.hub_name}"
  location            = azurerm_resource_group.hub.location
  resource_group_name = azurerm_resource_group.hub.name
  sku                 = "Standard"

  copy_paste_enabled     = true
  file_copy_enabled      = true
  ip_connect_enabled     = true
  shareable_link_enabled = false
  tunneling_enabled      = true

  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.hub_bastion[0].id
    public_ip_address_id = azurerm_public_ip.bastion[0].id
  }

  tags = local.common_tags
}

Spoke Virtual Networks

Create spoke VNets dynamically based on the input variable. Create spokes.tf:

# Resource groups for spoke VNets
resource "azurerm_resource_group" "spokes" {
  for_each = var.spoke_vnets

  name     = "rg-network-spoke-${each.key}-${var.environment}"
  location = var.location
  tags     = local.common_tags
}

# Spoke virtual networks
resource "azurerm_virtual_network" "spokes" {
  for_each = var.spoke_vnets

  name                = "vnet-spoke-${each.key}-${var.environment}"
  location            = azurerm_resource_group.spokes[each.key].location
  resource_group_name = azurerm_resource_group.spokes[each.key].name
  address_space       = each.value.address_space

  tags = local.common_tags
}

# Spoke subnets
resource "azurerm_subnet" "spoke_subnets" {
  for_each = merge([
    for spoke_key, spoke in var.spoke_vnets : {
      for subnet_key, subnet in spoke.subnets :
      "${spoke_key}-${subnet_key}" => merge(subnet, {
        spoke_key = spoke_key
        vnet_name = azurerm_virtual_network.spokes[spoke_key].name
        rg_name   = azurerm_resource_group.spokes[spoke_key].name
      })
    }
  ]...)

  name                 = "snet-${each.value.spoke_key}-${split("-", each.key)[1]}"
  resource_group_name  = each.value.rg_name
  virtual_network_name = each.value.vnet_name
  address_prefixes     = [each.value.address_prefix]
  service_endpoints    = each.value.service_endpoints

  dynamic "delegation" {
    for_each = each.value.delegation != null ? [each.value.delegation] : []
    content {
      name = delegation.value.name

      service_delegation {
        name    = delegation.value.service_delegation.name
        actions = delegation.value.service_delegation.actions
      }
    }
  }
}

# Network security groups for spoke subnets
resource "azurerm_network_security_group" "spoke_subnets" {
  for_each = azurerm_subnet.spoke_subnets

  name                = "nsg-${each.key}"
  location            = var.location
  resource_group_name = each.value.resource_group_name

  tags = local.common_tags
}

# NSG associations
resource "azurerm_subnet_network_security_group_association" "spoke_subnets" {
  for_each = azurerm_subnet.spoke_subnets

  subnet_id                 = each.value.id
  network_security_group_id = azurerm_network_security_group.spoke_subnets[each.key].id
}