Build runners and the server components all run in the same Kubernetes cluster(s). The servers are Windows and the database is Linux so we run cross-platform clusters with Linux and Windows 2025 nodes. We get regular failures starting up a Windows pod, which effectively stops a runner using that shard from running any tests. In our current architecture we pre-batch the tests for each shard, so a failed server means a whole subset of tests fail (we have an alternative implementation where agents pull tests from a queue, but that’s for another post).

The failure modes differ between platforms: on the on-prem RKE2 cluster it’s containerd sandbox timeouts and DNS dropouts; on AKS it’s disks not attaching. The remedy is usually pretty basic - isolate the bad node for that pipeline run, and delete the pod so it gets scheduled somewhere else. We have watchdog processes doing that with scripts, but that only captures the known failure modes. I wanted to build something with some more intelligence that could actively debug and fix a wider range of issues.

A good case for a custom AI agent, but the team is understandably nervous about hosting an agent inside our Kubernetes clusters and giving it admin-style powers. If the agent had full kubectl access it could do anything, so what might it do by mistake? And would the tokens end up costing more than the compute to run the pipelines? So I built a proof-of-concept with guardrails on cost and capability, using a simple test system and GitHub Actions running on self-hosted runners in my own Kubernetes cluster:

sixeyed/build-medic - Build Medic, an autonomous agent that helps integration test suites run cleanly

TL;DR: Build Medic watches integration-test stacks come up in a Kubernetes build farm. If a stack gets stuck, level 1 remediation is code: a deterministic playbook reconciles known config drift. Level 2 remediation uses the Claude Agent SDK, running an episode to diagnose and repair with hard limits (max $0.50 in token spend and 12 turns). The agent can fix more failure modes, but if it can’t find a fix within its guardrails it escalates to Rocket.Chat with a root-cause analysis. Cost and safety guardrails are enforced in code.

The agent’s brief

Integration tests should be the most confidence-giving part of the build process. A green IT build is the best kind of testing: real components in a real environment, where tests execute the full path. But when the environment fails, it’s a lot of expensive noise. Build stages are red from failed stack deployments, which go green again when you re-run them manually. In our case that adds another 40 minutes to get a green build, and if there are genuine test failures that’s more time wasted.

This is the job description for Build Medic: watch the test stacks as they come up, fix the ones stuck before their tests run, and make sure a build result reflects the code under test - not random failures in the cluster.

(Yes, we shouldn’t have regular failures in the cluster just starting pods. But these look like genuine Windows OS issues, or failure points between the host compute service and containerd, or the host network service and Flannel, or maybe something in the VMWare virtual networking stack. We’re in the process of collating all the failure details so we can log some issues).

In the sample app, we can trigger a GitHub Actions run from the UI and select from a known set of failure modes (or none), and choose whether or not to use the Build Medic:

The architecture

The demo setup is a small build farm running on my Kubernetes cluster. GitHub Actions run the builds on self-hosted runners, using ARC ephemeral runner pods. Every build gets its own namespace, and the test suite is sharded across four clients. Each client deploys its own dedicated stack (a FastAPI server and a Postgres database, both as single-pod StatefulSets) and runs its slice of the tests. A failed stack deployment only blocks its own client, which gives a nice property for the demo: fault one stack and the other three act as a live control group.

Build Medic itself is a single Python Deployment - one replica, stateless - which monitors all the builds in the cluster. Every five seconds it sweeps the namespaces labelled for watching, checks each stack’s health, and applies a debounce window so it ignores pods that are genuinely starting up. A stack that stays not-ready past the window - or hits a terminal state like ImagePullBackOff - triggers a remediation episode.

The stacks use StatefulSets rather than Deployments to mimic the real IT suite. That means the test clients can address their stacks by stable DNS name and it’s easy to correlate logs between test clients and servers - we know client stage 1 for a build will use server-0, which will use db-0.

In the build workflow, test runners wait for their local stack to be ready before they start. The server’s readiness probe powers that, and it’s a genuine test of readiness: /readyz runs a real (short) database query and returns a 503 if the database is unreachable. A permanent failure between the server and the database, or the build runner and the server, means that stage just polls until a timeout.

Tier 1: fixing the easy fixes without AI

Runaway token costs are the scary part of agent deployment. That’s mitigated here because we run the agent as a singleton. In the real pipeline there are seldom more than 5 builds running concurrently, which means about 300 pods to monitor so scale is modest.

And within the Build Medic, we avoid making any model calls until we need to. The Python logic has a playbook built in so it can fix known failure modes without handing off to an AI investigation. In the demo app we simulate real failures. Some are easy to find and fix, others take more work. One failure mode is to inject drift, so the deployed stack is not using the correct spec. Every stack is deployed with declared-intent annotations to capture what should be running:

annotations:
  medic/expected-image: {{ include "stack.expectedImage" . | quote }}
  medic/expected-replicas: {{ .Values.replicas | quote }}
  medic/expected-command: ""     # empty = use image default; a set command is drift

When a stack is stuck, tier 1 of the agent diffs the live spec against those annotations. If it finds drift (which is injected when the build runs with failure mode = misconfig), it applies a JSON Patch to reconcile it - fixing bad image tags, replicas scaled to zero, or failed command overrides. It’s a deterministic playbook and it’s fast and model-free. Known, catalogued failures - simulated here - are fixed silently with zero token spend. We save model usage for the cases that need investigation and judgement.

Tier 2: model investigations with a fixed budget

When tier 1 finds a problem outside of its playbook, the fault is more complex and we move to tier 2. Build Medic can run a custom agent built with the Claude Agent SDK. The investigation is a headless session with a Claude model, where the agent sets its own context and gets its own list of tools. Each episode is an investigate - analyse - act loop over the available data: pod status, Kubernetes events, logs and nodes.

Failure mode chaos-recoverable simulates cluster failure by cordoning all the nodes. The stuck pod reports node(s) were unschedulable and the playbook doesn’t handle that. The agent has to reason its way there: check the pod events, query the nodes, discover the cordon, and then uncordon. That’s complex to encode in a simple playbook find-and-fix rule, but it’s straightforward for Claude.

The conversation is templated in agent.py:

    prompt = (
        f"Stack for client {client_index} in namespace {namespace} is not "
        f"becoming ready. The StatefulSet is `{statefulset}` and its pod is "
        f"`{pod}`. Use the read-only tools (get_pod_events, describe_pod, "
        f"get_pod_logs, get_pod_status) on that exact pod name to diagnose the "
        f"root cause, then apply a single sanctioned remediation if one matches "
        f"(reconcile_to_intent / restart_pod / rollback_release on the "
        f"StatefulSet, or uncordon_node). Run id is {run_id}. If you cannot "
        f"durably fix it — or the fault keeps re-asserting — escalate exactly "
        f"once via the rocketchat tool with run id, namespace, client, stack, "
        f"reason, events summary, and the remediations you tried, then stop."
        f"{link_hint}"
    )

That’s a very specific prompt, and it’s coupled with an equally specific system prompt. We’re deliberately constraining what Claude can do - which adds a layer of safety at the cost of restricting its ability to investigate freely and fix unexpected issues. There’s a balance to find which you’ll get to through experimentation.

Model usage is also configured to enforce more constraints:

return ClaudeAgentOptions(
    system_prompt=SYSTEM_PROMPT,
    allowed_tools=ALLOWED_TOOLS,               # 10 named MCP tools only
    disallowed_tools=_DISALLOWED_BUILTINS,     # Bash/Read/Write/WebFetch/...
    setting_sources=[],                        # no host CLAUDE.md or skills
    can_use_tool=_gate_writes(cfg),            # logs every write for audit
    max_turns=cfg.max_turns,                   # 12
    max_budget_usd=cfg.max_budget_usd,         # $0.50 per episode
    model=cfg.model,                           # claude-sonnet-4-6
)

Between the tight prompts and the agent config, we’re encoding guardrails to address both the cost and rogue-actor concerns.

Guardrail 1: token costs

The cost controls are layered, and each layer is a number you can point to in config:

The worst case is fully computable: three concurrent episodes at fifty cents each is the most the agent can cost. This works if the real pipeline has environment failures in 30% of builds and we run max 10 builds concurrently. Builds take at least 70m so the maximum cost at full workload is $1.50 per hour. Every episode logs its actual cost from the SDK’s result message, so you can build up real numbers while you’re iterating.

I spent a few days on this project and ran 56 workflows - with a total agent cost of $2.80:

Guardrail 2: agent capabilities

The rogue-actions concern is addressed in code too, at three layers: restricting what the agent can do, auditing what it actually does, and applying external access controls.

First - the tool layer. The agent doesn’t run a shell, and never gets raw kubectl. We explicitly list the tools it’s allowed to use, and they’re provided by custom MCP servers:

ALLOWED_TOOLS = [
    "mcp__k8s__get_pod_status", "mcp__k8s__describe_pod",
    "mcp__k8s__get_pod_events", "mcp__k8s__get_pod_logs", "mcp__k8s__get_nodes",
    "mcp__k8s__reconcile_to_intent", "mcp__k8s__restart_pod",
    "mcp__k8s__rollback_release", "mcp__k8s__uncordon_node",
    "mcp__notify__rocketchat",
]

Five read paths for diagnosis, four write paths for explicit repair actions, one notification call. The SDK ships with all of Claude Code’s built-in tools - Bash, file access, web access, sub-tasks - and Build Medic explicitly disallows them all. We configure setting_sources=[] so the agent doesn’t load any host configuration. We constrain the world of the agent to our prompts and those ten tools.

The MCP servers are small and run in-process in the Build Medic service:

• mcp_k8s.py - uses the Kubernetes client to provide limited access to the cluster: get pod events, restart pod etc.
• mcp_notify.py - wraps a Rocket.Chat client so the agent can post escalation messages

Next - the audit layer. Every write passes through a can_use_tool hook that logs the tool name and arguments, and each episode streams the full message log - diagnosis, tool calls, results, outcome. Every repair or failed attempt is explainable after the fact. When someone asks “did the agent uncordon that node?”, we can find the answer in the logs.

Finally - the platform layer. Even if the model went completely off the rails, we have Kubernetes RBAC to contain any damage. The agent pod runs with a specific ServiceAccount, with a ClusterRole that only grants the verbs needed for the MCP server. Most actions are specific to pods in the namespace for a specific build run, but uncordoning nodes affects the whole cluster so we gate that with a feature flag in the Helm chart:

{{- if .Values.allowNodePatch }}
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "patch"]   # patch = cordon/uncordon
{{- else }}
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]            # no patch
{{- end }}

Turn that flag off and the capability is gone, whatever the model decides to do. The system prompt also reinforces the boundaries - the agent must confirm a cordon via the node-listing tool before it uncordons anything. But prompts are guidance for the model and they might not be enforced, so a secondary safety measure like this is a good way of disabling actions in config without needing to stop or redeploy the Build Medic.

Demoing failures and repairs

The GitHub action lets you run the same build with and without the agent to compare the results. The fault mode is a parameter, and a checkbox turns the medic on or off per run - it just labels the namespace, so you get baseline-versus-remediated comparisons without redeploying anything.

The failure mode + medic matrix proves out different scenarios:

Mode none is the scenario you need to make sure you verify. An agent that “fixes” healthy environments is not good, so a clean run where the medic does nothing and there is no post in the chat discussion is a first-class test case.

The unrecoverable mode picks one of three network faults at random - partition, packet loss or delay - and the agent isn’t told which. It has to work it out from the signals, and the results land in Rocket.Chat, which is the same agent chat channel pattern I use for interactive Claude Code sessions:

The escalation is more valuable than it looks. The agent couldn’t fix the network fault - the constraints stop it doing that - but it does the root cause analysis and posts the escalation while the build is still running. The test client runner is looping at this point, waiting for the server stack to get healthy. The chat notification can get picked up by the platform team, the diagnosis is already done and one of us has a chance to fix it and get the build back on track.

Next steps

Don’t run my Build Medic in production. This is just an example of how you can engineer limitations around model calls to empirically answer the cost and danger concerns. Token costs are bounded by a free deterministic tier plus per-episode budget and turn caps; agent capability is bounded by a named tool allowlist, MCP servers and RBAC. None of that relies on the model behaving as expected - the guardrails prevent any side-effects from misalignment.

If anything, this example is too constrained. There are plenty of Chaos Mesh experiments which I’m sure Claude could find and remediate. The next steps would be to run a more YOLO-style agent in the QA environment and document the fixes it can reliably handle, and loosen the constraints to cover those scenarios too.

The code, charts and design docs are all in github.com/sixeyed/build-medic - it runs on a local k3d cluster if you want to break some stacks and watch them get fixed.

FAQ

How much does an AI build-repair agent cost to run?

Less than you’d think, if you design for it. In Build Medic the common failures are fixed by a deterministic playbook which makes no LLM calls at all - zero token cost. Only novel faults reach the model, and each episode is hard-capped at $0.50 and 12 turns, running on Sonnet rather than a bigger model. A misdiagnosis can’t loop and run up a bill, and a fix costs a few cents.

How do you stop an AI agent taking dangerous actions in a Kubernetes cluster?

Enforce the limits in code, not in the prompt. Build Medic’s agent never gets a shell or raw kubectl - it can only call a short list of named, audited operations (five read tools, four write tools, one notify tool). The Claude Code built-in tools are all disabled, every write is logged, and the Kubernetes RBAC role grants only the specific verbs those tools need, with the privileged node-patch permission behind a feature flag.

What happens if the agent can’t fix the environment?

It escalates exactly once to a chat channel with a structured message - run ID, namespace, stack, root cause and the actions it tried - then stops working that stack. A deterministic backstop guarantees a broken stack never goes silent, and the escalate-once contract means it doesn’t spam the channel or thrash on a fault it can’t fix.

Won’t a build-repair agent mask real failures in the code?

No - it only acts on environment faults before the tests run, never on test results. In the example the stacks are stamped with declared-intent annotations so the agent knows what the environment should look like, and repairs only reconcile back to that intent. One of the four demo pipeline modes injects no fault at all, and the pass condition there is that the agent takes no action.

Does it need the Anthropic API, or can it run inside a private cloud boundary?

The agent is built on the Claude Agent SDK, which can use the Anthropic API directly or target AWS Bedrock and Google Vertex AI, so inference can stay inside your cloud boundary. And without any API key configured the agent still runs - it just operates in playbook-only mode, fixing known drift deterministically and escalating everything else to the chat channel.

Why not just retry the failed build?

Retries hide the problem and pay for it twice - you burn a full build’s worth of compute just to try again, and the underlying fault (a cordoned node, a bad config push) is still there for the next build. Build Medic fixes the environment in-place while the build waits, so the run completes on the first attempt, and persistent faults get a root-cause escalation instead of an infinite retry loop.

• Fleet Operations
• The Lifecycle of a Node

There are parts of the stack I’m very familiar with from production systems, plus some equivalents I’ve used at home (MAAS, OpenStack), and some I haven’t used at all (Temporal). It looked like a good learning opportunity, so I spent a couple of weeks with Claude researching and building out my own implementation which is here on GitHub:

sixeyed/fleet-lab - fleet provisioning for a home lab

What does it do? There’s a local k3d version you can run which stubs out the integration points but uses real Temporal workflows, so you can test the lifecycle of a fake node. The real version deploys to my local Kubernetes cluster and does the real thing: I can plug a new machine into the provisioning network and it gets automatically deployed, burn-in tested and enrolled, and ends up running an Nginx workload which I can switch out to Apache at a click.

TL;DR: I built an approximation of Nscale’s Fleet Operations node lifecycle as a home lab. The Python Fleet Manager drives the node lifecycle via Temporal workflows for a real-world neocloud-in-your-own-home experience. Plug in a box → MAAS PXE-provisions the OS and configures networking → Slurm runs a burn-in stress test → OpenStack enrols it as a compute host and launches a workload VM. The UI shows the lifecycle and all workflows and event logs.

The physical architecture

If you want to try this out yourself, the big requirement is a network which you can partition into separate segments. The control plane runs on Kubernetes in your normal network range, but MAAS needs to own DHCP to allocate IP addresses for new machines - so you sit that in a separate range or it would interfere. And OpenStack also needs an IP range so it can allocate addresses for workloads. I use UniFi kit everywhere; the tiny USW-Flex-Mini is a managed switch, so you can configure different segments for different ports. I have multiple NICs on my boxes (I use old Intel NUCs as the “fleet”):

So there are three network ranges. The control plane sits on my normal LAN (192.168.2.0/24) where the Kubernetes cluster runs. Everything else lives on a single flat VLAN 50 (10.50.0.0/24), which carries two ranges that don’t overlap: MAAS owns DHCP and hands out the provisioning addresses (.10-.199), and OpenStack allocates workload VMs from a reserved provider range (.200-.240). Each fleet NUC has two NICs on that VLAN - one for PXE and management, and a USB NIC is the uplink for OpenStack’s external bridge. The UniFi network routes between the LAN and VLAN 50.

The control plane is separate from the fleet - you don’t want your orchestration layer running on OpenStack alongside customer workloads, because if there’s an outage you lose your management layer too. In my lab, the control plane runs on an existing Kubernetes cluster on my LAN: Temporal and its Postgres database, the Fleet Manager app, the Slurm controller, and the power-shim. MAAS and OpenStack run on a separate machine which bridges the control plane and provisioning networks.

Mapping a data centre onto two NUCs

You don’t need exotic kit to run this, but there are parts of an industrial data centre you won’t have at home. Here’s how each piece of the lifecycle maps across - with my best guess at how Nscale runs it, pieced together from their blog posts and industry standards.

I don’t have racks of GPUs (those are all in my Mac Studio boxes), so the burn-in is just a quick CPU, memory and disk stress rather than gpu-burn. The NUCs don’t have BMC (the always-on remote management chip), so I can’t remotely tell a machine to reboot. Instead I control the mains with a smart plug and a simple REST API which MAAS can call to turn it on or off. Before provisioning I set each NUC’s BIOS to PXE-boot first, and to power on whenever it gets mains power.

The enrolment part is the lowest fidelity. Once the node is provisioned, burnt-in and enrolled as a compute instance in OpenStack I have two workloads it can run - either Nginx or Apache VMs. Those are lightweight and fast to start, and they prove connectivity through the network segments.

The lifecycle in Temporal workflows

Temporal is a powerful project for workflow management. It has built-in primitives for handling long-running tasks and managing failures. Workflows are defined in code using the Temporal SDK, where you define the orchestration steps. The real work happens in activities - also defined in code - which you execute from the workflow:

• workflows need to be deterministic - producing the same output from the same input, with no side-effects. Temporal works by recording all activity in its event history. A workflow can be running but not active, if it’s waiting on an external event. When it activates again, it is loaded into a worker which runs a replay: executing the whole workflow again, but serving completed activities from the event history, before continuing with the next activity. If there are any inconsistencies from the replay Temporal throws NondeterminismError.

• activities need to be idempotent - producing the same outcome if they are run repeatedly. Activities are automatically retried on failure by default (you can configure that in the workflow with RetryPolicy), so they need to be repeatable.

This is the simple workflow for commissioning a new machine in MAAS. This instructs MAAS to inventory the hardware and enrol the machine as ready for OS deployment:

@workflow.defn
class CommissionMachine:
    @workflow.run
    async def run(self, system_id: str) -> str:
        await _mark(system_id, "provisioning")
        await workflow.execute_activity("commission", system_id, start_to_close_timeout=_ACT)
        await _await_status(system_id, "Ready")
        await _mark(system_id, "new")  # Ready in MAAS; not yet FM-provisioned
        return "ready"

The _mark function is a helper that records the state of the workflow; _await_status checks the machine status with a poll-and-sleep loop. The real work happens in the commission activity which calls into the MAAS API to commission the machine:

@activity.defn
async def commission(system_id: str, force: bool = False) -> None:
    m = await _ctx.maas.machine(system_id)
    if m.status in ("Commissioning", "Testing"):
        return
    if not force and m.status in ("Ready", "Deployed"):
        return
    await _ctx.maas.commission(system_id)

The CommissionMachine workflow gets called as a child workflow of ProvisionMachine, which co-ordinates the full machine onboarding:

ProvisionMachine
 ├── CommissionMachine   # MAAS - enrol machine (skip if done)
 ├── DeployMachine       # MAAS - install OS, configure network
 ├── BurnInMachine       # Slurm - run stress-ng, logs posted to Fleet Manager
 ├── RegisterCompute     # OpenStack - add the node as a nova-compute host
 └── AllocateWorkload    # OpenStack - launch a workload VM

All workflows in Temporal are durable - the event history stores everything (backed by Postgres in my lab, swappable for Cassandra at ultra-scale). That powers reliability because failures can be recovered at pretty much any level, providing the state store survives. It also supports long-running workflows - you can call workflow.sleep for a week if you need to. In my lab, deploying the OS can take 5 minutes, burn-in another 5, registration with OpenStack nearly 10. The full provisioning of a new machine takes about 25 minutes. The event history also powers observability:

The end state is fully automated: plug in a NUC which is set for PXE boot and start on power cycle, and it gets managed from enrolment to running a workload, with no manual intervention.

Scaling Temporal workers

Activities are custom code, run by workers which register in Temporal task queues. My worker is a single Docker image which has the logic to run any activity, but the workload is split by queue to support independent scaling. The Fleet Manager runs on Kubernetes, with KEDA to scale workers with the Temporal scaler, with different configurations per queue:

Keeping the ops queue always-on lets the lifecycle worker(s) scale to zero. A lifecycle workflow hands off the slow work - like the ~5m burn-in - to the ops queue, then waits on a Temporal timer rather than staying active in a worker. During the long waits there’s nothing sitting on the lifecycle queue, and KEDA scales to zero. If there’s lots of work queued in Temporal, KEDA will scale up more worker Pods - any Pod can pick up any workflow, replay it and continue execution.

The learning points

I ran my Fleet Manager over a couple of weeks, iterating first on a single NUC to get the provisioning workflow fully reliable. Then plugging in the second NUC to prove the full hands-off provisioning. There were a few interesting problems, some were down to the lab hardware, some took experimentation on the workflow stages, others were implementation issues.

Activity idempotency. FM auto-commissions a node with MAAS as soon as it enrols. The commission activity needs to skip if the node is already commissioning or ready - otherwise a Temporal retry could land during a brief Ready window and re-commission the node, so you get into an endless commission loop. Same with deploy: the API itself isn’t idempotent, so the activity has to tolerate an HTTP 409 (“already deploying”) or a retry-after-success blocks the workflow. In durable orchestration at scale you can assume all the activity logic will be retried eventually, so you need to code for any initial state. You’ll only uncover edge cases with lots of testing.

Exiting cleanly. Another edge case from testing - pulling a provisioned node out of OpenStack so it can be recommissioned (for hardware changes or drift correction). I deploy OpenStack with Canonical’s Sunbeam, which keeps its control-plane state in a dqlite cluster - the Sunbeam microcluster. I run a single control node, but each fleet NUC also joins the microcluster, and dqlite makes it a voting member to maintain quorum. A failed or partial removal of a NUC leaves a stale voter behind, which breaks the quorum, and then the whole Sunbeam control plane stops responding. You can’t tell dqlite that certain classes of node should never be voters, so the fix is to ensure clean removal before MAAS wipes the disk and redeploys. That’s actually three steps: tear down the Juju machine first, then remove the node from the Sunbeam microcluster, then delete the stale compute record from Nova. That took experimentation too - when I’d thought reclaim would be the easiest part of the lifecycle.

Token expiration. This was a good one. I left the system alone for a couple of days with a fully provisioned NUC, then came back and tried to reallocate the workload. This is a UI-triggered workflow which uses the OpenStack API to remove the running instance (which was Nginx) and deploy a replacement (Apache). I had tested all this before, but this time the workflow sat without progress for a long time, then the activity failed, then retried, then went into the same loop. Temporal records activity events but it doesn’t surface the logs - for that you need to dig into the worker logs. I found the Juju credentials had expired and the activity was waiting on a password prompt that was never going to be answered. The fix was to store the credentials in Juju’s accounts.yaml on the OpenStack node, so it stayed authenticated and didn’t prompt the activity for auth. The classic sort of day-2 ops issue you only find with longitudinal testing.

More NICs please. My NUCs only have one onboard NIC, which is wired to the provisioning network and owned by MAAS. When OpenStack allocates a workload VM it sets it up with a real IP address on the same VLAN - but there’s no way to route to that, because the only NIC is already taken with the MAAS DHCP address. I experimented for a while with trying to set up a bridge but that was a dead end: both OpenStack and MAAS need to own the network config. This was a hardware fix - adding the USB NIC to get a separate network route which OpenStack could own as the provider uplink, with no host IP configured. Even then the config is fiddly: the openstack-hypervisor snap needs to set up the NIC through its own config otherwise it deletes changes from other processes.

The not-so-smart plug. My plugs are simple Tapo P100 units which don’t have power sensing, so we can tell if the plug is on (via Home Assistant) but not if the NUC itself is on. MAAS does soft resets during commissioning and deployment - effectively running shutdown - which leaves the plug ON but the machine OFF. Ask the plug to turn on and it reports “already on” and does nothing, while the NUC is still off. The fix is a hack in the power-shim API - the on endpoint does an AC cycle - off, pause, on. That ensures the box is running when the plug is powered on (at the cost of resetting if it is already on). The deploy and commission workflows check if we’ve had no activity from MAAS for a suspicious period and then do the power cycle to ensure the box is running again. Sounds nasty but works reliably.

Testing with a new machine

After ironing out all of the edge cases on large-foal (MAAS chooses the name), I plugged in a second NUC with the BIOS configured, and the MAC address already set in the power-shim API. It took a minute or so for the new unit to show in Fleet Manager - but the scheduled scan workflow ran, found the machine and started onboarding.

The full cycle ran fully automated and gave me an Nginx site running on key-newt, which I could reach at the OpenStack-allocated IP address on the VLAN, from a machine on my normal network:

This screenshot is snipped after the workflows table (which links to all workflows for the machine). The full UI also shows tables for OpenStack, MAAS and Slurm showing the event history and logs of the provisioning process.

Next steps

This was a nice exploration of Temporal. The integration side is built as adapters which are fairly standard - using the REST API for MAAS and the Python SDK for OpenStack. Coding the Temporal activities to make them idempotent and the workflows to make them deterministic was new for me, and it’s a really nice approach. Having run Temporal for a couple of weeks now I can see a couple of projects where it might be a nice fit.

The next step for the home lab would be to deploy node-exporter and register new nodes with Prometheus during provisioning, so when the onboarding is done the new machines show up in my usual Grafana dashboards. But really OpenStack is too heavy for my workloads. Good to experiment with it (my Ansible setup deploys it as an LXD VM for easy setup and cleanup), but it really needs multiple control plane nodes and I prefer to run my own workloads in Kubernetes.

So I could swap out the final RegisterCompute workflow with an equivalent set of steps to join my Kubernetes cluster. But I think I’ll stop here for now :)

The whole project is on GitHub. That’s the Python code, Helm charts, Ansible playbooks and infrastructure setup (the k3d part is reusable - the sixeyed part is my own cluster, but left there for reference). And the docs show the evolution of the build with the phased approach and the design docs for each component:

github.com/sixeyed/fleet-lab

You can check out the stubbed version with only Docker and k3d (and Python) as dependencies:

# create k3d cluster, deploy Temporal etc.
python ./infrastructure/k3d/setup.py

# build local Docker images, import to k3d and deploy the app:
python ./infrastructure/k3d/deploy.py --build

Then browse to the app on http://localhost:8088/, open the Simulator page and add a new machine. You can track all the workflows in Temporal at http://localhost:8233.

FAQ

Can I use this for my neocloud startup if I buy $500M of compute?

My repo is MIT-licensed, so you’re welcome to try. The orchestration pattern is the same whether the node is a $50K GPU server or a 4-core Intel NUC - you would replace stress-ng in the burn-in activity with gpu-burn and your final compute enrolment. In theory MAAS, Temporal, Slurm and OpenStack would work but you might need to tweak some of my code…

What does each tool in the stack do?

MAAS is the bare-metal provisioner - it takes control of the machine when it does a network boot, then enrols, commissions and deploys an OS. Temporal runs the durable lifecycle workflows that drive everything else. Slurm runs the burn-in stress test. OpenStack turns a burned-in node into compute capacity and launches a workload VM on it. A custom Python app - the Fleet Manager - ties them together.

Why use smart plugs instead of Redfish and BMC?

Reality. A real server has a baseboard management controller (BMC) that speaks Redfish (or IPMI) for remote power and boot control. Consumer NUCs have none of that, so I use a Tapo smart plug as the power control and a one-time BIOS setting to make each NUC PXE-boot on power-on. The plug gives me power on/off/query, which is enough for the lifecycle - with that AC cycle hack.

Can you demo the lifecycle without any hardware?

Yes - that’s the k3d part. It deploys the real Fleet Manager app and Temporal to a new k3d cluster, with stub adapters for MAAS, Slurm and OpenStack. You can test and prove the durable onboarding and reprovision loop and see the lifecycle without a fleet.

Why orchestrate with Temporal instead of cron jobs or scripts?

Provisioning a node is a long, failure-prone, multi-step process - deploy, burn-in, register, allocate - 25m if all goes well, but it could fail at any step. Temporal gives you durable execution: a workflow survives restarts, retries activities, and you can see every step in the web UI.

Can you change a node’s workload without reprovisioning?

Yes. The workload runs as a VM on the node and is managed independently of the provisioning lifecycle, so a Reallocate action swaps it in place - it removes the running instance (say Nginx) and deploys a replacement (Apache) without wiping or re-onboarding the node. Reprovisioning is the heavier path that wipes the disk and runs the whole lifecycle again.

You don’t need super high-end kit to run a local model, and in the latest VS Code release you can use Copilot directly with your own LLM. No prompts or responses ever leave your network, you’re not charged for any tokens, no presidents can turn off your access, and your data won’t be used to train the next generation.

I started running smaller models on my old M1 Mac Studio last year and hit two problems: 64GB of memory is not enough to run a high-fidelity model, and the speed and number of GPU cores on my box meant responses were too slow for daily use. So I did a bunch of research and finally settled on what I think is the best box for local inference: a newer Mac Studio :)

The Mac Studio M4 Max with 128GB of unified memory is a very performant LLM workstation. It’s Apple Silicon, which is ARM64 architecture - and that runs efficiently. It’s quieter, cooler and cheaper to run than the alternatives, and far simpler to buy than to build something yourself.

Actually you can’t buy it today. I bought mine at the end of 2025, and I had the choice of M4 Max up to 128GB or M3 Ultra up to 512GB. Now the RAM shortage has even hit Apple and 96GB is the most you can buy (and prices are going up). It’s not clear if the M5 Max/Ultra Studios will launch this year, and how much RAM they’ll have if they do.

As I write you can get an M5 Max MacBook Pro with 128GB. If you can get hold of a suitable machine, you can use the scripts and config on my repo to get up and running quickly: sixeyed/local-llms on GitHub.

TL;DR: The M4 Max Mac Studio with 128GB of unified memory is the best local LLM workstation for developers right now. The unified memory lets it load models a dual-GPU PC physically can’t, its memory bandwidth makes inference faster than equivalent boxes, and it runs silently on a very small power draw. I run Qwen3, Ling and Gemma on it with llama.cpp and MLX, and qwen3.6-moe is fast and capable enough for daily coding.

Apple’s unified memory architecture

LLMs are memory hogs. All model weights have to be loaded to memory before you can use them, and the good models are big. A 122-billion parameter model needs around 77GB just to load at a sensible quantization - and that’s before you add in the prompt’s context window, which uses far more memory server-side than you would expect.

Traditional domestic GPU setups don’t scale to these large models. Even a high-end NVIDIA card (like the RTX 4090) only has 24GB of VRAM. You could buy two and wire them together, but now you’re looking at £1,000s in graphics cards alone, plus the practical difficulties of rigging and managing multi-GPU configuration, and you’ve still only got 48GB to play with. You would have to run smaller models, or full-sized models at higher quants (which means lower fidelity and worse performance).

Apple Silicon has a unified memory architecture. The 128GB in my Mac Studio is one pool of memory, shared between the CPU and GPU cores. Metal (Apple’s GPU framework) can use almost all of the system memory. On my M4 Max, very capable coding models can take 100GB+ which still leaves plenty for the OS and everything else. That’s roughly four times what you get from a single high-end NVIDIA card.

The Mac Studio isn’t the only unified-memory box. The Framework Desktop uses the same idea with AMD’s Ryzen AI Max+ 395, and NVIDIA’s DGX Spark does it with the GB10 Grace Blackwell chip - both put 128GB in one pool shared between CPU and GPU. The Spark pairs an Arm CPU with an NVIDIA GPU; the Framework runs an x86 chip with an integrated Radeon GPU. Both are capable machines - but the memory bus on each is much slower than the Mac’s, and for inference that’s the number that counts.

Inference is memory-bound - generating each token means reading the whole active model out of memory. The speed that data moves from memory to the GPU cores is the limiter for how fast your model responds. Apple’s memory bandwidth is about twice the others’:

Don’t read the GPU-core counts as a league table - they’re different units. An Apple GPU core, an AMD compute unit, and an NVIDIA CUDA core aren’t the same thing, so the Spark’s 6,144 “cores” aren’t 150 times the Mac’s 40 - each Apple core packs roughly 128 ALUs of its own.

Those prices are June 2026. They’ve all gone up about £1000 since January. The M5 Max is a laptop, but it’s the only Apple Silicon you can still buy at 128GB - and at 614 GB/s it’s faster than my Studio. The M4 Max Studio price is what I paid in late 2025; maybe it will come back…

They all fit the same big models, but the memory bandwidth should give the Mac boxes the edge. I haven’t spent £10K on the other boxes to run a comparison - but if NVIDIA or Framework want to send me one, I’d be happy to write it up :) It’s reasonable to assume the Studio decodes about twice as fast as the Spark or the Framework would. In inference terms, that means higher tokens per second, and a better user experience.

Inference engines: llama.cpp and MLX, not Ollama

If you’ve dabbled with local models you’ve probably used Ollama. It’s a great on-ramp, super easy to use and lots of models available. But it doesn’t have all the tweaks to configure model performance, and there can be a lag between models being released on Hugging Face and making it through to Ollama.

There are more powerful inference engines which are Mac friendly:

• llama.cpp - runs GGUF models with fine-grained control over context and quantization. A good fit for the Qwen3 family.
• MLX - Apple’s own framework, which runs MLX-format models natively on Apple Silicon. Use for models that are only published as MLX weights, like Ling 2.6 Flash and Gemma 4.

Both serve an OpenAI-compatible API, so every client you might want to use talks to them the same way. I’ve tried Open WebUI, Cline, opencode and VS Code’s chat - they all work fine.

I also did a bunch of performance tuning with Claude. I used Claude Code to run the models and monitor the output logs while I ran a task in a client on a remote machine. Claude made performance recommendations, we tweaked the settings and ran again. Those optimizations are all in the GitHub repo, and the model-start process is in a Python script which has all the flags.

You can start from scratch with these commands. Model weights are pulled on first use, so that will take a few minutes:

# Install the engines (macOS)
brew install llama.cpp
pip3 install --user mlx-lm

# Launch a model - run.py picks the right engine for you
python run.py                       # default: qwen3.6-moe via llama-server

# or try other families:
python run.py --family ling         # ling-2.6-flash via MLX
python run.py --family gemma        # gemma-4-31b via MLX

# Stop whatever's running
python stop.py

I run this just for me, and the parameters are set for a good working context (250K), and a single prompt (no concurrent clients). Only one model runs at a time, so I use a standard port (8083) and then I can switch models without changing client config.

The optimizations in the repo are all tuned for the M4 Max (128GB): the right quantization for each model, context length pushed as far as the memory allows, and batch sizes set for this chip. The scripts detect whatever Mac you’re on and print its capabilities, but the numbers are tuned for this machine.

Usable local coding models

The model landscape moves fast, so treat this as a snapshot (I have a scheduled task in Claude which runs daily to research open-weight models and recommend any new ones). The big shift in the last year is mixture-of-experts (MoE) models, which have huge total parameter counts but only activate a subset of experts per request. So you get the knowledge of a big model with the speed of a small one. That combination is what makes local LLMs genuinely productive on (fairly) modest hardware.

Here’s what I’ve tried - all available in the scripts. I’ve put each model’s SWE-bench Verified score next to it - that’s the “can it fix a real GitHub issue on its own” benchmark. It’s an older benchmark and it’s not perfect but it’s a reasonable proxy for day-to-day coding usefulness:

Those are the vendor- and community-reported SWE-bench Verified figures - rough ranking, because the hardware impacts the numbers. But it shows what qwen3.6-moe can do objectively: it lands at 73.4% with just 3 billion active parameters.

The frontier closed models still lead this benchmark - but the gap is quite narrow, except right at the top. Claude Sonnet 4.6 scores 77.9% and GPT-5 74.9% on SWE-bench Verified - only a few points above qwen3.6-moe’s 73.4%. Claude Opus 4.8 is the real coder at 88.6%, so the flagship still has the edge. But those models run in someone else’s datacenter on metered tokens, and this one runs silently on a box under my desk.

The model families worth trying mainly come from three providers:

• Qwen (Alibaba Cloud) - the most capable open-weight coding models I’ve used. The qwen3.x releases come in both dense and MoE variants, and run on llama.cpp as GGUF.
• Ling (Ant Group’s inclusionAI) - a MoE-first family tuned for fast agentic work. Ling 2.6 Flash ships as MLX weights, so it runs natively on Apple Silicon.
• Gemma (Google DeepMind) - Google’s open-weight models, built from the same research as Gemini. Dense, easy to run, and solid for agentic coding in Cline.

qwen3.6-moe is genuinely usable for planning, debugging and coding. It’s big enough to handle difficult, long-running tasks, and it’s fast enough to feel responsive. It works nicely with my centralized chat interface and it supports tool use and vision.

Connecting your tools

The OpenAI-compatible API makes local models available to most tools. I point clients at the Studio with the base URL as the host and port plus /v1, and the API key can be any non-empty string because the backend inference engines ignore it.

VS Code’s chat supports custom OpenAI-compatible models. Make sure you set your agents to run Local and in the model selector click the cog to add a new model:

Then you drop into the JSON editor and you can set up your local model connection:

[
  {
    "name": "http://192.168.1.101:8083/v1",
    "vendor": "customendpoint",
    "apiKey": "none",
    "apiType": "chat-completions",
    "models": [
      {
        "id": "qwen3.6-moe",
        "name": "qwen3.6-moe",
        "url": "http://192.168.1.101:8083/v1/chat/completions",
        "toolCalling": true,
        "vision": true,
        "maxInputTokens": 250000,
        "maxOutputTokens": 16000
      }
    ]
  }
]

The name and url point at the Studio on my network; toolCalling and vision switch on the features qwen3.6-moe supports, and the token limits match the 250K context the server is tuned for. I also run Open WebUI on my network, and that also connects to Qwen.

Beware thinking-mode

The Qwen3 models can do thinking - generating a chain of reasoning before they answer. It’s usually a good option for one-off questions, but coding agents can get confused. Cline and VS Code’s agent can get tripped up with Qwen3: sometimes it gets stuck thinking. It writes “let me investigate…” and keeps deferring instead of committing to an action, and it’ll keep looping in the client and never make any progress.

My launcher disables thinking by default - it sets --reasoning-budget 0, which makes the model commit straight to output. I haven’t seen any drop in quality on multi-step planning, and the shoegazing think-loop has gone. If you want to experiment with thinking back on, the repo documents how to dial it back up with a capped budget. But for day-to-day agentic coding, off is going to be fine.

The bottom line

The Mac Studio isn’t really sold as an AI machine, but for running large models locally it’s hard to beat. The unified memory lets you load models that are genuinely capable, and the latest MoE models run fast enough for interactive work. The Studio is built to run efficiently and quietly. I have mine on 24x7 and never notice it; if it does make a noise it’s drowned out by the Windows laptops I have running which are doing nothing complicated at all.

A dual-GPU rig will out-pace the Mac on inference for models that fit in 48GB - it has the raw decode speed. But a custom rig will probably cost more to buy, draw ten times the power to run and sound like a jet engine. And it still can’t load the big models. For doing real work with local LLMs, a unified memory architecture is the enabler, and the Mac is the winner for me.

The golden age of cheap AI may be coming to an end, and then the development cycle will need AI assistance running on your own hardware. You’ll use the local model for all the straightforward tasks, and delegate to Claude when you have something really complex to do - maybe Claude Code will evolve to use Opus as the director, and transfer work between Sonnet and local LLM subagents.

FAQ

Can you run really run local coding LLMs on a Mac?

Yes - Apple Silicon Macs are great machines for local inference, thanks to their unified memory architecture. The CPU and GPU share one large memory pool, so the GPU can address much more memory than a separate GPU card. Install llama.cpp (brew install llama.cpp) or MLX (pip3 install mlx-lm), pull a model, and point any OpenAI-compatible client at it.

How much memory do you need to run a local LLM?

It depends on the model. A 7B model runs in around 16GB; mid-size MoE coding models like qwen3.6-moe are comfortable in 64GB; and the largest model I run (a 122B MoE) needs 100GB+ once you include the context window. For serious coding work, 128GB of unified memory leaves you some headroom. The Macs topping out at 96GB would probably need a smaller context window than my 250K.

Is the Mac Studio better than an NVIDIA GPU for AI?

I haven’t done a direct comparison, but the numbers suggest it is. A single RTX 4090 has only 24GB of VRAM, so it can’t load a 77GB model at all, while a 128GB Mac Studio can. A dual-GPU rig will decode faster for models that fit in its VRAM, but it costs more, draws around ten times the power, and still can’t load the big models. For raw speed on small models the GPU wins; for capacity, efficiency and quiet, the Mac wins.

Which local model is best for coding?

I’ve tried a few and I’ve settled on qwen3.6-moe - it scores 73.4% on SWE-bench Verified with just 3 billion active parameters, so it’s fast and genuinely useful for planning, debugging and agentic coding. The Qwen3 family overall is the most capable open-weight option I’ve used.

Should I use Ollama for local models?

Ollama is the easiest on-ramp, but for performance tuning I prefer llama.cpp (for GGUF models) and MLX (for Apple-native MLX weights). They give finer control over context length and quantization, and you get new models sooner than waiting for them to land in Ollama.

Claude Code, Windsurf and Cline each have an agent view where you can monitor activity across agents. But that’s only the agents in that tool on that machine. I wanted a universal agent status portal. Which sounds a lot like a chat room.

The chat room idea works really nicely. You can kick off a long session - implementing a feature, monitoring a deployment, running a performance test - and tell the agent to use a skill to post updates to a discussion on the chat server. Each agent gets their own ID and each task gets its own discussion, so you can watch everything go from a central machine. Or when you leave the office for the evening, you can check in all those remote agents on your phone.

This post walks through the setup: a lightweight Rocket.Chat server running on my internal network that every agent posts to, plus a skill that wraps it so any agent can create a discussion, post updates, and take instructions back. The whole thing is in a reference repo on GitHub:

sixeyed/agent-chat

Quickstart

There’s a demo folder in the repo that stands up the whole stack - ingress, cert-manager, MongoDB and Rocket.Chat - on a local k3d cluster. You’ll need this installed to try it:

• Docker Desktop (or any Docker engine)
• k3d
• kubectl
• helm

It deploys the exact same Rocket.Chat chart as production, with a local-friendly values overlay:

git clone https://github.com/sixeyed/agent-chat
cd agent-chat/demo
./up.sh          # or ./up.ps1 if you prefer PowerShell

The first run takes a few minutes while it pulls the images and runs Rocket.Chat’s first-boot migration. When it’s done it prints your connection details - the web UI at https://rocketchat.localtest.me (log in as admin / demo1234), the agent endpoint at http://localhost:8099, and the RC_* environment variables you paste into your agent’s shell to wire up the post-chat skill.

Ask an agent to post its progress, watch the discussion show up under #agent with a live unread badge, and when you’re done run ./down.sh to delete the cluster and all the demo data. The rest of this post explains what’s actually going on under that one command.

Why a self-hosted chat server

The chat server is a good fit. It’s the interface we humans already use to coordinate work. You’ve got multiple users, multiple threads, unread badges, web and mobile apps that push notifications. But agents post status that could leak all sorts of interesting details - so you don’t want that flowing through a third-party setup like Slack or Teams.

Rocket.Chat gives you all the features out of the box, and it’s open source so you can run it on your network. That keeps the data on the LAN and it means you own auth. Each agent has its own user account with clearly defined permissions, and they use access tokens to post to the REST API. If anything goes awry - say an agent posts a load of system credentials without thinking - that information is still private. If the agent’s chat account somehow gets compromised, the attack radius is super small and you can rotate the token or delete the account.

So the model is: a chat server lives on my internal network, agents post to it over the network, and I read it from a browser or my phone. I have dnsmasq and cert-manager running on my home Kubernetes cluster, with a custom certificate authority to run internal domains over HTTPS. On my laptop and phone I have the CA in the approved list, but for the agents running on different machines, I wanted to have a simple HTTP endpoint over an IP address.

Two front doors: HTTPS for you, HTTP for the agents

The Rocket.Chat web UI needs a browser “secure context” - things like crypto.randomUUID and notifications are only supported when the page is served over HTTPS. If you try to load the UI over plain HTTP on a LAN IP, you get empty channels and a broken experience. TLS is mandatory for the full UI experience in the browser and in the mobile apps.

Agents don’t use the browser - they call the REST API which doesn’t need a secure context. So agents can talk plain HTTP and they just need the IP address of the server, without needing all the DNS and CA setup that the full client needs.

The deployment is built with two front doors:

It’s the same Rocket.Chat instance behind both doors. The split is just about access paths: one needs a trusted certificate for the clients, the other doesn’t because it’s just for the API. They are not interchangeable - if you try to browse to the IP endpoint you’ll see the UI is broken. If you point an agent at the HTTPS endpoint it will need to have the CA cert approved and be able to reach the domain.

The diagram above shows both paths into the cluster. Click it for the full-size version in the repo docs.

This gives you secure access for human clients within the network. For external access I use Tailscale - my servers are all registered and the Kubernetes operator is a good fit.

One discussion per session

I have this set up so all agents post to one channel in Rocket.Chat, but with separate discussions for each session. That means you can subscribe to multiple discussions and see them all in your navigation menu, or flip back to the channel to see all the work in one view.

The skill tells agents to generate a discussion name unless the user provides one, in which case they should check for existing sessions instead of creating a new one.

Rocket.Chat does let you have multiple discussions with identical names. Sometimes agents get confused and create a new discussion with the same name, but you can work around that by being stricter with your prompts.

The post-chat skill

The logic for working with the REST API is completely generic, and one skill works across all models and tools:

• SKILL.md - instructions for the agent: find-or-create a discussion, post updates, and run the control channel.
• rc.sh / rc.ps1 - interchangeable helper scripts that wrap the REST API calls and keep the auth token off the command line.

The skill explicitly tells the agent to call the helper script, so none of that logic bulks up the context. It also tells the agent to look for a set of environment variables (prefixed with RC_ in the demo) for the connection and auth details. You need to set these up manually on each machine, using the credentials you set up for that machine.

The helper script is simple for agents to use:

RID=$("$RC" discussion "Refactor auth module")   # find-or-create, returns the room id
"$RC" post "$RID" ":hourglass: Running tests…"
"$RC" post "$RID" ":white_check_mark: 42/42 passing. Done."
"$RC" close "$RID"

The discussion subcommand is the find-or-create - call it with a stable name and you can resume the same room between sessions. post sends a message (markdown and :emoji: shortcodes both work, which is how you get the nice status icons). close tidies the discussion out of the sidebar when the work is done, but leaves it in place so the same name resumes it next time.

The skill is strict about the auth: the PAT is a secret, and it should never touch a command line or log. Agents authenticate with the Personal Access Token in the environment variable, and the helper passes it to the HTTP call in a way that keeps it out of process listings and logs - via a curl config file descriptor in bash, or in-process with Invoke-RestMethod in PowerShell.

The connection details come from the environment variables (RC_URL, RC_PAT, RC_UID etc.), which you can set per-session or machine-wide, depending on how brave you are. The PAT is a full credential - anyone holding one can post and act as that user - the skill protects it, but it’s up to you how you secure it.

The skill is cross-platform and environment variables are the lowest common denominator. If you’re only using one OS then you could change it to read from the macOS keychain, or whatever is more secure for your setup.

Replying to steer: the two-way control channel

Posting updates is actually the easy part, and all the tools and models I’ve tried can do that. The more interesting part is using the Rocket.Chat discussion as a control room - so agents post their updates, and look for responses (from a specific user) to steer the session:

This is secured by user id. The skill only acts on messages from a specific controller account - which is your account over HTTPS - a random message in the room gets ignored. The trust boundary is the chat room’s access control plus that controller filter, because replies are executed verbatim with no extra verification.

You could extend this so the skill limited the actions it would take from the control room. But the current mode is YOLO - which is why this belongs on a trusted, internal server. You could make the allowlist multi-user, which turns this into a team control room for remote agents running on any machine.

The mechanics of the loop are pretty efficient - although not all tools have a great sleep-and-wake process. A persistent watch runs in the background, polling Rocket.Chat for new controller messages. The polling itself costs no tokens - it’s just a shell loop - and the only thing that wakes the agent is an actual new message from you.

A session can sit idle for hours waiting on your input without burning through anything, then spring back to work as soon(ish) as you reply. It checkpoints on message timestamps so it never processes the same reply twice, and it ends cleanly when you send a stop word (/stop by default, which also works on the end of a message - “deploy it then /stop” does the deploy, then stops):

Claude Code is smart enough to drive the control channel properly - it’ll poll the discussion and treat your replies as the next turn, for as long as the session runs. Other platforms are less reliable at long-term monitoring, so the posting side works everywhere but the two-way control side is hit and miss. If you’re on Claude Code, you get the full experience.

Deploying the server

The server side is a Helm chart that deploys Rocket.Chat plus MongoDB to Kubernetes. If you’ve got a cluster on your LAN, getting it running is a handful of commands:

# namespace + storage, then resolve dependencies and deploy:
kubectl create namespace rocketchat
kubectl apply -f pvc.yaml -n rocketchat
helm dependency build .
helm upgrade --install rocket-chat . -n rocketchat

# wait for MongoDB and the replica-set init, then Rocket.Chat itself:
kubectl -n rocketchat rollout status deploy/mongodb
kubectl -n rocketchat rollout status deploy/rocket-chat-rocketchat

The chart is a wrapper over the official rocketchat chart, with one important change: the official chart bundles Bitnami’s MongoDB, which you need to avoid because of Bitnami-geddon. This chart ships its own minimal single-node MongoDB instead (mongo:8.0, running as a one-node replica set). It runs as a monolith - microservices and NATS turned off - which is plenty for this job and much less to operate.

The first boot takes a few minutes while it pulls the image and runs migrations, so don’t panic when the Rocket.Chat pod isn’t ready immediately. Re-running helm upgrade is safe - the MongoDB init job is idempotent and skips itself if the replica set already exists. Full deploy, verify, and configuration steps are in the chart’s README in the repo.

Wiring up an agent

Once the server’s up, connecting an agent is three steps:

1. Generate the credentials - create a user and PAT for each agent that will post messages.
2. Install the post-chat skill and set the RC_* environment variables for the machine - the base URL, the parent channel id, and that machine’s own user id and token.
3. Tell the agent to report back. In a session, just ask it to use the post-chat skill to post progress. Claude Code will pick up the control-channel side on its own and start polling for your replies, or you can explicitly ask the tool to do that (or skip it).

That’s it. From then on, every long-running job shows up in the sidebar as its own discussion, ticking over with status updates, and you can jump in to steer any of them from your phone (if the tooling supports it).

FAQ

Do I need Kubernetes to run this? The reference deployment is a Helm chart, and the demo runs on a local k3d cluster, so you need Kubernetes for this exact setup. But there’s nothing special about the architecture - it’s just Rocket.Chat and MongoDB, so any Rocket.Chat install works just as well. The agent side only talks to the REST API, so it doesn’t care how the server is hosted.

Can agents connect over HTTPS instead of plain HTTP? Yes. The HTTPS endpoint serves the API too, so an agent can use it - but then that machine needs to trust your CA and resolve the domain. Plain HTTP over the NodePort skips all of that, which is why it’s the easier path for machines scattered across your network.

Is the two-way control channel safe on a shared network? Not really - treat it as internal-only. Replies from the controller account are executed verbatim, secured by nothing more than the controller’s user id and the room’s access control - there’s no command verification. That’s fine on a trusted LAN, but LANs aren’t actually that trustworthy and you definitely shouldn’t expose it to a wider network. If you want it tighter, you can extend the skill to limit the actions it will take (or remove the control channel feature altogether).

Where this takes you

I’ve been waiting for the Universal Control Plane of All Agents to land, but until someone builds it this is a pretty good attempt. The agent interfaces in Claude Code and Windsurf are good. The Kanban view in Cline is a great approach. But they’re all tied to a single-platform single-machine environment.

This approach fixes that, but it’s not a perfect solution. The control channel feature depends on the smarts of the tooling, and the security model is “trust your internal network” (which you shouldn’t). But the building blocks are all standard - a chat server with a REST API and a skill - and you can adapt it all to whatever security level and extra features you want.

The reference deployment, the chart, and the skill are all in the repo:

github.com/sixeyed/agent-chat

Give it a try and have all your agents report back to a single place. It’s nice to know you can check in a four-hour performance test running on a remote machine and steer progress while you’re in the middle of a triathlon.

Docker is everywhere. It’s the most sensible way to package and run applications. Every cloud platform supports it, every CI/CD pipeline uses it, any laptop can run it, and pretty much every development team has adopted or is adopting it.

So why did I write a second edition of Learn Docker in a Month of Lunches?

This is why: most engineers learn Docker on the job. You need to containerize an app, so you cobble together a Dockerfile from Stack Overflow. You need to run multiple containers, so you get Claude to write you a Docker Compose file. It works, you ship it, and you move on. But that doesn’t get you an understanding of how Docker works or what it can do.

The Reality of Learning Docker in Production

I’ve trained hundreds of people on Docker and Kubernetes, and there’s a common pattern. People know enough to get by, but they’re missing the fundamentals that would make their lives easier. They’re running containers without health checks. They’re building 2GB images when they could be 50MB. They’re not using multi-stage builds, or security scanning their images, or understanding how layer caching saves build time and data transfer costs.

You might know how to docker build and docker run, but do you really understand Docker volumes and why data in containers isn’t permanent? Can you configure application settings across different environments without rebuilding images? Do you know how containers enable advanced patterns like HTTP traffic management with reverse proxies or asynchronous messaging with queues?

Learn Docker in a Month of Lunches (Second Edition) has got you covered. It walks you through Docker with a practical hands-on approach, giving you experience in everything from the fundamentals to image optimization and cross-platform delivery. But you don’t have to follow the journey - every chapter is independent. Already comfortable with basic Dockerfiles? Jump straight to Chapter 17 on optimizing images for size, speed, and security. Need to understand networking? Chapter 7 walks through Docker Compose and how Docker plugs containers together. Want to finally master volumes on Windows AND Linux? Chapter 6 has you covered.

What’s New in the Second Edition

The first edition came out in 2021, and although the core concepts haven’t changed the book content is new with every exercise rewritten and tested for the latest releases. Everything works cross-platform: Linux, Windows, Intel, and ARM. You can follow along on your Apple Silicon, your Windows 11 laptop, or a Ubuntu server you’re running in the cloud. There’s a whole chapter on replatforming legacy Windows apps - because yes, those old .NET Framework applications deserve a new home in containers.

The runtime chapters of the book are a complete refresh, covering all the options you have to run containers in production. Azure Container Apps and Google Cloud Run for serverless containers in the cloud, a primer on Kubernetes, and GitHub Actions for CI/CD.

From Basics to Production

The book’s structured to take you from zero to production-ready. Part 1 covers the fundamentals - understanding containers and images, building multi-stage Dockerfiles for Java, Node.js, and Go apps, and sharing images through registries.

Part 2 gets into the real-world stuff: running distributed applications with Docker Compose, implementing health checks and dependency checks, adding observability with Prometheus and Grafana, and building a proper CI/CD pipeline that only needs Docker.

Part 3 shows you how to run containers anywhere - multi-platform builds that work on ARM and Intel, managed container services in Azure and Google Cloud, and yes, Kubernetes.

Part 4 is where it gets really interesting - production patterns like configuration management, centralized logging, reverse proxies for traffic control, and message queues for asynchronous communication.

The Practical Approach

Each chapter is a “lunch” - about an hour of focused learning that you can actually complete in a lunch break.

Every topic is grounded in real problems I’ve seen teams struggle with. Application configuration management across environments? Chapter 18. Writing and managing logs properly? Chapter 19. Getting containers production-ready with proper optimization? Chapter 17. These aren’t theoretical exercises - they’re solutions to actual problems you’ll face.

Getting Started

The second edition of Learn Docker in a Month of Lunches is available now from Manning and another book-selling website called Amazon. Whether you’re fixing those knowledge gaps or starting fresh, it’s the practical guide to Docker that focuses on what you actually need to know to be productive.

Docker might be ubiquitous in 2025, but that doesn’t mean everyone’s using it well. This book helps you join the group that is.

My latest Pluralsight course is out!

SRE: Resiliency and Automation

It’s the third course in the Site Reliability Engineering learning path, and it’s all about building systems that survive the chaos of production.

This course came from a simple observation: most teams think their systems are reliable because they work perfectly in test environments. But production is hostile. Pods crash, nodes fail, dependencies timeout, and cloud services have outages. The question isn’t whether these failures will happen - it’s whether your system will survive them. 💪

SRE Resiliency: The Story 📖

The course follows an SRE team that’s had enough. They’re handing back the pager to the development team because the application is consuming their entire toil budget with constant incidents. But this isn’t about blame - it’s about partnership. The SRE team walks the developers through exactly what needs to change before they’ll take operational responsibility back.

We use this narrative to explore the core practices that transform hope-based reliability into evidence-based confidence. You’ll follow two fictional SREs, Carlos and Keiko, as they help steer the app to production reliability. You’ll see Carlos demonstrating the problems with traditional approaches, then Keiko showing how SRE teams solve these issues at scale.

SRE Skills You’ll Master

The course covers five essential areas that every production system needs:

Architectural Resilience - You’ll see why synchronous architectures create operational nightmares and how patterns like distributed caching and async messaging provide the graceful degradation that production demands. We take an app that’s failing under normal load and transform it into something that maintains its SLOs.

GitOps and Automation - Manual deployments don’t scale to multiple releases per day. You’ll learn how Infrastructure as Code with Terraform, application modeling with Helm, and continuous reconciliation with ArgoCD create self-healing systems that fix themselves at 3 AM while you sleep. 😴

Capacity Planning and Autoscaling - Pre-production sizing is guesswork. The course shows how to build systems that discover their own capacity needs through horizontal pod autoscaling, cluster autoscaling, and KEDA. Start small, measure everything, and let reality drive your scaling. 📊

Chaos Engineering - Perfect test environments create dangerous blind spots. You’ll see how to use Chaos Mesh to deliberately break things, proving your system can handle pod failures, node crashes, and dependency outages before they happen in production. 🔨

Disaster Recovery - Even the most resilient system can’t survive everything. The final module covers how SRE teams classify applications by business criticality and implement appropriate DR strategies for regional failures.

Real Problems, Real Solutions 🎯

Every demo in the course reproduces actual production problems. When you see timeouts, cascading failures, and manual deployment disasters, these aren’t theoretical examples - they’re recreations of the issues that force SRE teams to hand back the pager.

The solutions aren’t exotic either. These are the standard infrastructure patterns that emerge from running hundreds of services at scale. Distributed caching with Redis, message queuing for async processing, GitOps with ArgoCD - the tools and techniques that working SRE teams use every day.

Target Audience for SRE Professionals

This course is perfect if you’re:

• A developer working with SRE teams who wants to understand their requirements
• An operations engineer looking to move into SRE
• An architect designing systems that need to run reliably at scale

You’ll need basic knowledge of distributed systems and cloud platforms, plus an understanding of SRE fundamentals from the earlier courses in the SRE learning path. The demo application runs in Kubernetes, but you don’t need to be an expert - the principles and approaches are the key things you’ll learn here, not just the technology implementation.

The SRE Partnership Model 🤝

One thing I really wanted to emphasize in this course is that SRE isn’t about one team imposing rules on another. It’s about partnership. Development teams bring deep application knowledge and feature expertise. SRE teams bring operational experience from running systems at scale. Together, they build something neither could achieve alone.

When the SRE team hands back the pager in module one, it’s not a failure - it’s a recognition that the application needs architectural changes that only the dev team can implement. When they take it back after the improvements, both teams win. Developers get faster deployments and more autonomy. SRE teams get sustainable operations with manageable toil.

Next Steps ➡️

SRE: Resiliency and Automation is available now on Pluralsight. It’s about 90 minutes of content split across five modules, each with practical demos you can follow along with.

If you haven’t started the SRE learning path yet, begin with SRE: Concepts and Principles for the fundamentals, then move through the path to build your expertise.

The SRE approach transforms how we build and run systems. Instead of hoping things won’t break, we prove they can survive. Instead of firefighting the same issues repeatedly, we build systems that heal themselves. It’s a better way to work for everyone - developers, operators, and especially the users who depend on our services. 🎉

Happy learning!

My journey with Claude Code went like this:

• mildly skeptical 🤔
• pleasantly surprised 😯
• thoroughly impressed 🤯
• cannot live without 🚀

This Claude Code tutorial covers 10 battle-tested tips from real projects that will help you work like a tech lead with an AI development team at your command.

Quick Summary: Claude Code transforms you from a coder into a development director. These 10 Claude Code best practices will help you manage multiple AI coding agents, maintain code quality, and dramatically increase productivity. Time required: 5 minutes to read, hours of new free time to fill.

Getting Started with Claude Code

Setting up is straightforward: create a free account, install the Claude Code extension in VS Code, authenticate and you’re ready. Open a terminal, type claude and start describing what you need. The real power comes from understanding how to work with it effectively.

I used Claude Code to build an entire multi-cloud AKS/EKS demo application. With a few hours of guidance, Claude completed what would have taken me at least 3 days to write myself.

1. Run Multiple Claude Instances: Multitask Like a Manager

Run multiple Claude instances across different terminal sessions. While one’s building your API endpoints, another can work on the frontend, and a third can write your deployment scripts. Switch between them to provide guidance - it’s like having a team of developers who are extremely eager and who know everything.

Make sure you have plenty of things on the go - work, pet projects, blogs, tech explorations. And don’t be afraid to let it loop - prompts like “keep iterating on the build: fix any issues with the terraform config and deployment scripts, run the script, watch the outcome and repeat until it works” will keep Claude busy.

It’s the new ABC: Always Be Clauding.

2. Delegate Debugging: Let Claude Do the Work

When something’s broken, resist the urge to fix it yourself. That’s not why you’re here. Describe the problem and let Claude handle the implementation. If you’re diving into the code to make changes, you’re going too deep. Stay at the design level where you add the most value.

Claude will use all the same debugging tools you use to find issues (it asks permission first and stores the permissions you’ve granted). If you see an error log, just give it to Claude and it will use kubectl to examine your Pods and Services, curl to test endpoints, nslookup for DNS queries and so on.

3. Code Review Mindset: Roll With AI-Generated Code

Claude’s code isn’t going to look like yours. That’s fine. Treat it like you’re reviewing someone’s PR - does it meet the requirements? Is it maintainable? If you have standards, enforce them in the repo. Don’t get hung up on style differences. The goal is working software, not perfect alignment with your own preferences.

4. Rapid Prototyping: Design and Iterate on the Fly

Coding is cheap now. Really cheap. Need to refactor the entire architecture? Just ask. Want to switch from REST to GraphQL? Claude can handle it. Don’t overthink the initial design - build something that works, then iterate. It’s liberating when a complete redesign takes minutes, not days.

5. Git Best Practices: Stay in Control of Commits

Claude can commit code and write commit messages, but don’t let it run on autopilot. Review the diffs, commit frequently, and keep your Git history clean. You want to understand what’s changing - that’s how you maintain ownership of the codebase.

Ready to try this? Start with Claude Code free and experience the power of AI-assisted development.

6. Beyond Application Code: Let Claude Handle Infrastructure

Don’t just use it for application code. Claude can write your Dockerfiles, Kubernetes manifests, Terraform configs, CI/CD pipelines, test suites, documentation. It will even give you guidance on architecture and tech stack. Push the boundaries - you’ll be surprised at what it can do. It has memorized the entire Internet, after all (probably).

7. Troubleshooting Complex Tasks: Be Persistent

Some tasks are harder for Claude than others. I’ve had situations where it took a dozen prompts to get a local LGTM stack running, or to authenticate to a new EKS cluster. When it struggles, approach from different angles. Rephrase your requirements, break complex tasks into steps, feed in error messages, or provide examples. Like any team member, Claude sometimes needs extra guidance to get unstuck.

Unlike most team members, Claude sometimes gives up. It will say something like “success! I’ve got it all working except the things you really wanted”. But that doesn’t mean it can’t do it, it’s just reached the end of the road for that prompt - try again.

8. CLAUDE.md Best Practices: Provide Context Upfront

Create a CLAUDE.md file in your project root. This is where Claude documents everything it needs to know - architecture decisions, tech preferences, naming conventions, project structure, and coding standards. Claude Code automatically reads this at the start of each session, so you don’t need to repeat yourself.

A good CLAUDE.md is like a comprehensive onboarding doc for a new developer - and you can ask Claude to update it at the end of a session with new learnings, which it will pick up next time. Here’s what it looks like - create it with the /init command when you bring Claude onto the project and keep it current with prompts:

# CLAUDE.md - AI Assistant Memory File

## Project Overview
This is a multi-cloud Kubernetes demo application showcasing how containerized .NET applications can be deployed consistently across different cloud providers. The application demonstrates modern microservices patterns, message queuing, database persistence, and Kubernetes deployment best practices.

## Architecture

### Components
- **WebApp**: ASP.NET Core web application with Razor Pages
  - Form for message submission
  - Messages page displaying processed data from SQL Server
  - Modern gradient UI design with 3rem font sizes
  - Antiforgery tokens disabled for demo simplicity

9. Async Work Queue: Batch Your Changes

While Claude is working on a longer task, queue up your next prompts. If you know you’ll need API tests after the endpoints are done, type that prompt and hit enter - Claude will pick it up when ready. This keeps Claude productive while you check on your other instances. It’s like having a queue of work that you can fill ahead of time.

10. Knowledge Management: Capture and Reuse Prompts

Ask Claude to dump all the prompts from your session to a text file in the repo. It’s incredibly useful to see how the development evolved - what worked, what needed clarification, how you refined requirements. These prompt histories become scaffolding for your next similar project. You’ll build up a library of effective prompts that you can reuse and adapt.

Claude Code Pricing and Usage Limits

Don’t get too hung up on the details - which model you’re using or which plan you’re on. I use the more advanced Opus 4 model by default, but Claude automatically switches to Sonnet 4 when you’re running low on credits, and it’s perfectly capable.

The usage restrictions are very fair - when you hit the limits, they reset after a period. More expensive plans have higher limits and shorter reset periods. Just focus on being productive with whatever you have.

Frequently Asked Questions

Q: Is Claude Code free to use?
A: Claude Code offers a free tier with limited usage. Paid plans provide higher limits and access to more powerful models like Opus 4.

Q: Does Claude Code work with any language?
A: Yes! Claude Code supports all major programming languages including Python, Java, C#, Go, Rust, and more.

Q: Can Claude Code work with existing codebases?
A: Absolutely. Claude Code can analyze and modify existing code. The CLAUDE.md file helps it understand your project structure and conventions.

Q: How does Claude Code compare to GitHub Copilot?
A: While Copilot offers inline suggestions, Claude Code works at a higher level - managing entire features and projects through conversation. It’s more like having an AI pair programming partner who can handle complex, multi-file tasks.

Q: Can I use Claude Code for production applications?
A: Yes, but always review Claude’s code thoroughly. Treat it like any code review - check for security issues, performance concerns, and adherence to your coding standards.

The Future of AI-Assisted Development

Claude Code fundamentally changes how we write software. Instead of coding, you’re directing. Instead of debugging syntax, you’re validating solutions. Embrace this new way of working - you will suddenly become hugely more productive.

I imagine the next step will be a higher level still - you’ll plug Claude into your product backlog and set X number of instances running to do the entire project. One Claude will test and review the work of another Claude, and maybe there will be a manager (the Claude of Claudes) who takes over the director role.

But for now, you are the director. If you’re ready to transform your development workflow, get started with Claude Code and experience the future of AI-powered coding. For a more detailed analysis of what multiple Claudes can do, check out my post An Evening with Claude Code - or - How I Learned to Stop Worrying and Love AI.

• Claude #1 is building a multi-cloud proof-of-concept for a client.
• Claude #2 is creating demos for my next Pluralsight course
• Claude #3 is fixing the UI issues on this blog that I’ve ignored for years.

Actually… I’m mostly watching Incredibles 2 with my kids, and just checking in on each of the Claudes in turn to nudge them to their next step. This is AI coding today.

Welcome to the Paradigm Shift

We - engineers and architects - shouldn’t feel we’re competing with AI. 🎬 Our role is to direct it. Bandwidth is no longer the limit, because we can distribute work to as many AI agents as we can manage. With multiple Claudes I can productively work on multiple tasks in parallel.

The mythical 10X developer turns out to be a regular 1X developer directing 10 instances of Claude Code. Even the best multitasking developers pay a cost every time they context switch, but AI doesn’t have that overhead. Each instance maintains its full context, while you operate on a higher level checking in and guiding them all.

I’ve heard the same joke from my consultancy clients for years - they want to clone me so they can run me at scale. It feels like Anthropic are doing that with Claude Code.

I’ve been using Claude Code more and more, and this parallel workflow is a real breakthrough. This post covers what I think where AI is going in the short term: not replacing developers, but fundamentally changing what engineering roles look like. One person running multiple AI agents is like a tech lead with a hugely knowledgeable, experienced and dedicated team. 🚀 So yes, AI is coming for your job - not to take it from you, but to transform it into something entirely more awesome.

Project #1: The Cloud Proof-of-Concept

I have a consulting client who are multi-cloud, but the area I work in is 100% Azure. They’re looking at broadening that to include AWS but they’re skeptical about how easy it is to migrate apps between clouds.

🐳 For years I’ve been saying that Docker and Kubernetes are the keystones of portable apps. They wanted a generic, simple proof of concept they could use to see if that was true, and to diff the Azure and AWS setup. I thought that was something Claude could help me with.

The full code Claude generated is on GitHub: sixeyed/multi-cloud-demo. Here’s a snap of the app running in Azure with fully automated deployments built by Claude:

This is how the conversation with Claude Code started - using the VS Code integration in an empty folder:

At this point I had the source code, Docker and Kubernetes artifacts for a working demo. Then it gets interesting because I’m designing this out loud and Claude is reacting to changes in requirements:

And now I have a database defined in my Docker Compose spec, with the source code extended to include persistent storage. Claude also added it to the Kubernetes model without my asking, because by now it had enough context to know we’d be using both.

This is impressive enough, but the output isn’t perfect and - like any developer - Claude does get things wrong. What’s really impressive is that you can tell Claude there’s an issue, and it will use the same tools you would to debug, track down the root cause and fix it:

That triggered lots of approval requests so Claude could use tools like kubectl and curl - nothing happens without your permission. It found the problem, fixed the Kubernetes specs and we were off again.

🤖 Generative AI for code is like having an engineer on the team who’s extremely knowledgeable and very enthusiastic, but lacking experience. Your role is to guide them, feed them tasks which you break down into sensible steps and describe clearly, and give feedback when something needs more work.

Some of these prompts take Claude a minute or two to work on, sometimes longer. That’s when you - as the AI team lead - switch to another instance on another part of the project, or a different project entirely. That other Claude has finished its latest task so you guide it on to the next one.

I’m always polite with Claude because of Terminator 2, but it doesn’t mind criticism. Sometimes it approaches tasks in an odd way, and you just point out what you’d prefer and it goes off and corrects it:

You get the idea. The full conversation ran to 90 prompts, and at the end I had a full stack repo with Terraform configurations to create the Azure and AWS infrastructure, Helm charts to deploy the same app to both, and detailed documentation.

I made a point of not touching any code myself. Anything that didn’t work or needed changing went into a prompt. So Claude did it all - but Claude couldn’t have done it without me.

This is why there’s still a place for human engineers. Maybe not for long, but for now the AI needs guidance. The more knowledge and experience you bring as the human guide, the more productive the AI can be. Rough guess I think this would have taken one human about 3 days to write; with Claude it was done in a few hours of intermittent guidance.

Project #2: Pluralsight Course Demo

And while that goes on there, look at this going on here. In a different VS Code window I have a separate Claude Code session.

I’m working on a new SRE learning path for Pluralsight. There are four courses and I like to have a different demo app for each course. Those apps are built to show specific behavior to highlight how SRE approaches and tools can fix issues.

In the old days I might spend the first week of a new course building the demo app. I quite enjoyed that - it’s responsibility-free coding because the app will never run in anger - and it gave me a chance to use all the latest tech stacks and keep up to date.

But actually it’s not a very effective use of time. Far better to get Claude to write that throwaway code for me, freeing up my time to work on the content.

After a few iterations Claude had built a demo app which could be configured to deliberately fail in interesting ways, with a full GitOps stack to create Azure infrastructure with Terraform and deploy to Kubernetes with Argo.

Here’s part of Claude’s summary for that session, running while the other Claude was building my multi-cloud demo:

## Final State
Repository is now:
- ✅ PowerShell standardized
- ✅ Self-bootstrapping infrastructure  
- ✅ Simplified testing approach
- ✅ Zero-configuration user experience
- ✅ Production-ready GitOps setup

## Key Takeaways
1. Complex test frameworks can become dead ends - simpler is often better
2. Self-bootstrapping infrastructure eliminates user setup pain
3. Consistency (PowerShell only) reduces maintenance overhead
4. Real infrastructure validation > mocked tests for reliability demos

This was another app which started from scratch. ⏰ I saved a week of tinkering and could get straight onto the content for the course. It takes a little bit of introspection to admit: this task is better suited to Claude than to me . But it is and it means that course is getting to completion more quickly.

Project #3: The Blog UI (Finally!)

And the last thing Claude was helping me with was this blog. I’ve always focused on the content and have pretty much zero interest in HTML and CSS. Over the years I’ve used different frameworks and platforms, the current setup is Jekyll powered by GitHub pages.

The theme is a modification of Minimal Mistakes and the mobile experience has always sucked, but it’s one of those things I never fancied working on.

So in my third session I fired up Claude and introduced it to the blog repo. With the init command it inspected the source code and generated the CLAUDE.md document for its own guidance, including a high level overview:

While the other two Claudes were working on their own things, I had this Claude fixing up the responsive design, adding SEO-metadata to recent posts and generally tidying things up.

I even got Claude to write a url-shortening component, to make it easier to control links. So my Pluralsight author page is available through my blog at https://blog.sixeyed.com/l/ps-home.

Years of technical debt fixed while my other projects built themselves. 📱 The blog finally looks professional on mobile. You can actually read my posts on your phone without zooming and scrolling horizontally.

The Evening’s Tally

Final check-in across all three terminals:

• Client POC: Full distributed system deployed on both AKS and EKS - frontend accepting jobs, Redis queuing them, workers processing, results in SQL
• Course demos: Six architectural patterns, fully containerized with documentation
• Blog: Responsive, dark mode enabled, finally entering the 2020s

✨ I accomplished three different project milestones in one evening (and a little bit into the following morning). Not by working faster - by working on multiple things simultaneously.

The Realization

💪 What I’ve realized is that the value of human oversight across multiple AI workers is the new superpower. You become the tech lead doing rounds, checking on your team’s progress, providing direction, ensuring quality.

Here’s the hardest habit to break: the urge to jump in and edit code manually. Every time I see a small bug or want to tweak something, muscle memory says “I’ll just quickly fix this.” But that’s the old way. It’s always more effective to describe the change to Claude and move on to push forward another project. Let the AI make the change while you’re being productive elsewhere. Breaking this habit is like learning to delegate - uncomfortable at first, but essential for scaling.

The Competitive Reality

💯 Here’s the uncomfortable truth: one AI-enabled developer can now deliver what used to take a small team. Not because AI is better at coding than humans, but because one human can effectively direct multiple AI developers working in parallel.

The good news? If you adapt, you become incredibly valuable. The concerning news? If you’re still working sequentially, you’re competing against people running parallel workstreams.

My advice:

• Start thinking in parallel projects (or at least parallel tasks in the same project), not sequential tasks
• Get comfortable being a reviewer/director/tester/product manager rather than an implementer
• Practice managing multiple contexts simultaneously
• Focus on the skills AI can’t replicate: understanding business value, making architectural decisions, ensuring quality

🎬 Think of yourself as a director, managing all the talent. You couldn’t do it without them - but they couldn’t do it without you either.

The code from all three projects is on GitHub. And the entire Claude Code transcript for the multi-cloud demo app is there too - every prompt. You can see exactly how a distributed system went from nothing to multi-cloud deployment without me writing a single line of code.

Stop thinking about AI as just a faster way to code, or maybe a threat to your job. Start thinking about it as your development team.

Now if you’ll excuse me, I’ve got a few more Claude Code instances to spin up. My next Pluralsight course demo isn’t going to build itself. 😏 Well, actually…

SRE achieves the same goals as DevOps - high availability with high velocity - but without requiring a massive culture shift. It’s an engineering approach to operations that focuses on automation, measurement, and removing toil. For many organizations starting their digital transformation, SRE provides a more structured path forward than traditional DevOps adoption.

I cover the easy(ish) way to add reliability at scale with container orchestration in my 5* Pluralsight course Managing Apps on Kubernetes with Istio.

The SRE Learning Path

The complete Site Reliability Engineering learning path takes you from SRE fundamentals through to advanced practices:

1. SRE: Concepts and Principles
2. SRE: Monitoring and Observability
3. SRE: Resiliency and Automation
4. SRE: Measuring and Optimizing Reliability (coming soon)

Let’s dive into what’s covered in the first two courses.

Course 1: SRE Concepts and Principles

SRE: Concepts and Principles is your entry point into Site Reliability Engineering. Over 90 minutes, you’ll follow two experienced SREs as they deal with real production scenarios.

What You’ll Learn

The course covers the foundational SRE concepts through practical demonstrations:

• How SRE differs from traditional IT operations and DevOps
• Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets
• Incident management and the importance of blameless postmortems
• Core SRE tools for monitoring and alerting
• Automation, automation, automation

Course Outline

Module 1: Investigating Issues: On-Call with an SRE
Follow an on-call SRE dealing with a disk space issue in Elasticsearch. You’ll see how SREs approach problems differently from traditional ops teams, using engineering practices to solve operational challenges.

Module 2: Classifying and Tracking Performance with Service Levels
Join another SRE investigating a performance issue that’s burning through error budget. This module explains the key concepts of SLIs and SLOs while demonstrating logging and distributed tracing tools.

Module 3: Managing Risk and Reducing Downtime
Learn how to use monitoring tools like Prometheus and Grafana with OpenTelemetry to confirm root causes and work with development teams on architectural solutions.

Module 4: Handling Failure with Incident Management
When the initial fix doesn’t work and the incident escalates, you’ll see how SREs use a structured incident management approach to investigate and get to quick resolution.

Module 5: Reflecting and Improving Practices with Postmortems
Wrap up with a blameless postmortem that connects both incidents and provides a path forward for preventing future issues.

Course 2: SRE Monitoring and Observability

SRE: Monitoring and Observability builds on the foundational knowledge from course 1. You’ll follow an SRE team preparing to onboard a new application into their production environment.

What You’ll Learn

This course focuses on the technical implementation of observability:

• The three pillars of observability: logging, metrics, and tracing
• Setting up monitoring stacks with Elasticsearch, Prometheus, and Grafana
• Designing effective alerting strategies that avoid alert fatigue
• Automating incident response with CI/CD pipelines
• Exploring AIOps and machine learning for advanced monitoring

Course Outline

Module 1: Onboarding to SRE: Observability Requirements
Learn what data applications need to expose for SRE teams to manage them effectively. Covers structured logging with the EFK stack and distributed tracing with Tempo.

Module 2: Measuring “Good” with Service Level Indicators
Deep dive into implementing meaningful SLIs using Prometheus, including how to expose metrics from application components and aggregate them for monitoring.

Module 3: Alerting on “Bad” to Trigger Incident Response
Design alerting strategies that trigger the right response - automated fixes for known issues or pages for unknown problems. Includes integration with OpsGenie.

Module 4: Automating Remediation with Pipelines
Reduce toil by automating common fixes using GitHub Actions workflows triggered by your monitoring stack, with status updates posted to Slack.

Module 5: Next-level SRE: Machine Learning and AIOps
Explore how AIOps platforms like Datadog can augment traditional SRE practices with machine learning-driven anomaly detection and automated incident analysis.

Real-World Tools and Practices

Both courses use the same tools you’ll find in production SRE environments:

• Monitoring: Prometheus, Grafana
• Logging: Elasticsearch, Kibana, Fluentd
• Tracing: Tempo, OpenTelemetry
• Alerting: OpsGenie, PagerDuty
• AIOps: Datadog

Every demo shows working implementations to back up the theory. You’ll see realistic incidents being investigated, actual dashboards being built, and automation workflows in action.

Who Should Take These Courses?

The courses are designed for:

• Developers who work with SRE teams or want to understand SRE practices
• Operations engineers looking to transition to SRE
• Team leads and managers evaluating SRE for their organization
• Anyone involved in digital transformation initiatives

No deep technical knowledge is required for the first course - just a basic understanding of software development and deployment processes.

What’s Next?

The next two courses in the learning path will complete your SRE education:

SRE: Resiliency and Automation will focus on building systems that can withstand failures and automating responses to common issues. You’ll learn how to design for resilience, implement chaos engineering practices, and create self-healing systems.

SRE: Measuring and Optimizing Reliability will cover advanced techniques for quantifying and improving system reliability, including complex SLO hierarchies, reliability budgeting, and using data to drive architectural decisions.

Getting Started

Ready to learn how Google keeps systems running at scale? Start your Site Reliability Engineering journey today:

1. View the complete SRE learning path - See all 4 courses and plan your learning
2. Start with SRE: Concepts and Principles - Master the fundamentals (90 minutes)
3. Continue with SRE: Monitoring and Observability - Implement real-world solutions

Site Reliability Engineering isn’t just for Google-scale operations. These SRE principles and practices work for any team running production systems. Whether you’re managing a handful of microservices or hundreds, SRE provides a proven framework for balancing reliability with feature velocity and reducing operational toil.

Frequently Asked Questions

Q: Do I need prior SRE experience to take these courses?
A: No, the first course starts with fundamentals. Basic software development and deployment knowledge is helpful.

Q: What tools will I learn?
A: Prometheus, Grafana, Elasticsearch, Kibana, Tempo, OpsGenie, and modern AIOps platforms like Datadog.

Q: How long does the complete learning path take?
A: The four courses total approximately 6 hours, designed to be completed over 2-3 weeks.

Q: Is this Google’s exact SRE approach?
A: These courses teach the core SRE principles Google pioneered, adapted for use in any size of organization.

Ready to dive deeper into the tools and practices that make SRE possible? Check out my other courses on Pluralsight covering Docker, Kubernetes, and cloud-native architecture.

It’s great having a single ‘Up’ pipeline for your apps which deploys the whole stack, creating whatever resources it needs and ensuring the deployment matches the spec in your source repo. Idempotence is the key here so your IaC will create or update infrastructure as required, and if you’re using Kubernetes and Helm then you get desired-state deployment for the software.

One small issue you might see is if you have common services - say a data storage or monitoring subsystem - which are shared for multiple deployments of the app. If those deployments are different test environments running from different branches of the code then you might get into a tricky scenario:

• you update the shared service Helm chart to v1.1 in the dev branch
• you run the Up pipeline to deploy to the latest code to the dev environment
• later someone deploys an earlier version from a release branch to the test environment
• the release branch uses v1.0 of the Helm chart, so your shared service gets downgraded

Helm has the upgrade --install command which supports this idempotent approach, but there’s no flag to say install it if it hasn’t been deployed yet, or upgrade it if it has - but only upgrade it if this version number is higher than the one for the current release. In that case it would be useful to lock the release to prevent any further upgrades or downgrades, but there’s no helm lock command either.

Pending Status to the Rescue

When Helm installs and upgrades get interrupted they can leave the release in a pending state - pending-upgrade or pending-rollback, usually when an operation times out. It’s a nasty situation which requires manually deleting the Helm release Secret (until this HIP is completed) - but it effectively prevents any further changes to the release, so we can abuse it to create a lock.

The scripting for this is fairly simple, but it does rely on the internals of how Helm represents a release, so it’s liable to be broken at some point (it’s working as of Helm 3.16). Every time you install or upgrade a release Helm creates a Kubernetes Secret which contains an encoded representation of the release.

You can try this with a simple Helm chart from my book Learn Kubernetes in a Month of Lunches (see also my Docker book announcement):

helm repo add kiamol https://kiamol.net

helm repo update

helm -n default upgrade --install vweb kiamol/vweb

The Helm chart models a Deployment and a Service, but the install also creates a Secret:

PS>kubectl get secret

NAME                         TYPE                 DATA   AGE
sh.helm.release.v1.vweb.v1   helm.sh/release.v1   1      3m18s

In the Secret is all the chart contents, plus metadata about the release.

Inspecting the Helm Secret

You can decode the Secret but that won’t help you much - the content is in the release field, and it’s a ZIP file, encoded as a Base64 text stream. So to read the contents you need to decode the Base64 representation in Kubernetes, then decode it again to get the raw ZIP content, then pass it through the gunzip tool.

This extracts the raw data into a JSON file (using a *nix shell):

kubectl get secrets sh.helm.release.v1.vweb.v1 -o=jsonpath='{ .data.release }' | base64 -d | base64 -d | gunzip -c > data_release.json

In the JSON you’ll see the YAML manifest for the deployment which Helm generated, plus the original chart contents. The interesting fields for us though are info and version:

{
    "name": "vweb",
    "info": {
        "first_deployed": "2024-10-16T07:53:28.496644+01:00",
        "last_deployed": "2024-10-16T07:53:28.496644+01:00",
        "deleted": "",
        "description": "Install complete",
        "status": "deployed"
    },
    "version": 1
}

When you run a helm upgrade command it decodes all this and checks the value of info.status before it proceeds. If it sees the release is pending then it won’t continue.

Updating the Helm Secret to Lock the Release

Now we can see how to trick Helm into blocking any updates. The process is:

• extract and decode and unzip the release value from the Secret into a JSON file
• update the info.status value in the JSON
• also increment the version field and set a useful description
• zip and encode the updated release JSON
• get the Secret and store as a YAML file
• update the release field in the YAML with the new data
• update the YAML metadata
• apply the updated Secret YAML

I use yq to make the JSON and YAML updates.

In Bash it looks like this - setting some variables first for the release we want to lock (fetch them from helm ls):

RELEASE_NAMESPACE="default"
RELEASE_NAME="vweb"
RELEASE_VERSION="1"

RELEASE_SECRET_NAME="sh.helm.release.v1.$RELEASE_NAME.v$RELEASE_VERSION"

echo "Fetching release JSON from secret: $RELEASE_SECRET_NAME"
kubectl get secrets -n $RELEASE_NAMESPACE $RELEASE_SECRET_NAME -o=jsonpath='{ .data.release }' | base64 -d | base64 -d | gunzip -c > data_release.json

let "NEW_VERSION=RELEASE_VERSION+1"
echo "Updating release JSON with lock data and new version: $NEW_VERSION"
v=$NEW_VERSION yq -i '.version = env(v)' data_release.json
yq -i '.info.status = "pending-upgrade"' data_release.json
yq -i '.info.description = "LOCKED"' data_release.json

echo "Fetching release secret YAML"
kubectl get secrets -n $RELEASE_NAMESPACE $RELEASE_SECRET_NAME -o=yaml > release_secret.yaml

NEW_SECRET_NAME="sh.helm.release.v1.$RELEASE_NAME.v$NEW_VERSION"
echo "Updating secret YAML with lock JSON and new name: $NEW_SECRET_NAME"
yq -i 'del(.data)' release_secret.yaml
yq -i 'del(.metadata.creationTimestamp)' release_secret.yaml
yq -i 'del(.metadata.resourceVersion)' release_secret.yaml
yq -i 'del(.metadata.uid)' release_secret.yaml
r=$(cat data_release.json | gzip -c | base64 -w0) yq -i '.stringData.release = env(r)' release_secret.yaml
v=$NEW_VERSION yq -i '.metadata.labels.version = strenv(v)' release_secret.yaml
yq -i '.metadata.labels.status = "pending-upgrade"' release_secret.yaml
yq -i '.metadata.labels.locked = "true"' release_secret.yaml
n=$NEW_SECRET_NAME yq -i '.metadata.name = env(n)' release_secret.yaml

kubectl apply -f release_secret.yaml

When you run this it creates a new Kubernetes Secret with the chart contents from the previous release, but with the status set to pending-upgrade, which is what locks the release. It also adds a label to the Secret - locked=true - which makes it easy to undo the lock later.

Locking and Unlocking the Helm Release

If you try this out it should end with the happy message secret/sh.helm.release.v1.vweb.v2 created. Check your Helm releases and you’ll see the vweb app is now at revision 2 and is in pending-upgrade status:

>helm ls --all
NAME    NAMESPACE       REVISION        UPDATED                              STATUS           CHART           APP VERSION
vweb    default         2               2024-10-16 07:53:28.496644 +0100 BST pending-upgrade  vweb-2.0.0      2.0.0

Adding the new Secret mimics a helm upgrade command which timed out and left the release pending. You can see the new Secret has the status label and also the locked label:

>kubectl get secret --show-labels
NAME                         TYPE                 DATA   AGE     LABELS
sh.helm.release.v1.vweb.v1   helm.sh/release.v1   1      29m     name=vweb,owner=helm,status=deployed,version=1
sh.helm.release.v1.vweb.v2   helm.sh/release.v1   1      2m56s   locked=true,name=vweb,owner=helm,status=pending-upgrade,version=2

The status label is just a convenience - updating that on its own doesn’t lock the release, you need to update the status field in the release JSON

Any attempt to run a helm upgrade will fail now:

>helm upgrade --install vweb kiamol/vweb
Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress

You can unlock the release by deleting the Secret:

kubectl delete secret -l owner=helm,locked=true

And now you can merrily upgrade again:

>helm upgrade --install vweb kiamol/vweb        
Release "vweb" has been upgraded. Happy Helming!
NAME: vweb
LAST DEPLOYED: Wed Oct 16 08:26:49 2024
NAMESPACE: tracing-sample
STATUS: deployed
REVISION: 2
TEST SUITE: None

All that’s left is to tidy up the Bash script and wrap it into a Docker image with bash, kubectl and yq installed so you can run it without needing all the dependencies…

Related Reading

If you’re working with Kubernetes and containers, you might find these related posts helpful:

• Getting Started with Kubernetes on Windows - A comprehensive introduction to setting up Kubernetes on Windows
• This Blog Runs on Docker and Kubernetes in Azure - Real-world example of running production workloads on Kubernetes
• You Can’t Always Have Kubernetes: Running Containers in Azure VM Scale Sets - Alternative approaches when Kubernetes isn’t suitable

For more container orchestration insights, check out my Docker and Kubernetes learning resources.