I kept watching this happen. Ten minutes of exploration before any real work, every single time. If you work across multiple repos or come back to a project after a couple weeks, it’s worse. The agent is essentially reading the codebase for the first time, again.

I wanted to fix this.

The idea

A few weeks ago Karpathy tweeted about using LLMs to build personal knowledge bases. The workflow: collect raw sources, have an LLM compile them into a structured wiki of markdown files, then query and build on that wiki over time. Every query makes the wiki richer. The knowledge adds up.

The part that stuck with me: he’s not using fancy RAG. The LLM maintains its own index files and summaries, and at his scale (~100 articles, ~400K words) it just works. The LLM reads its own compiled knowledge to answer questions.

Codebases are raw data too. Source files are unstructured information that happens to be executable. What if the LLM compiled a codebase into a wiki the same way, with module overviews, architecture docs, concept articles, and then used that wiki as its starting point for every session?

That’s CodeWiki.

How it works

CodeWiki is a thin Rust CLI called cw paired with a Claude Code skill. The CLI handles git ops, directory scaffolding, and metadata. The agent does all the actual reading and writing. No API keys, no LLM calls from the CLI. Your agent is the intelligence.

When you run cw init in a repo, it creates a wiki directory at ~/.codewiki/<project>/ with this structure:

~/.codewiki/my-project/
├── _index.md         # master index
├── _architecture.md  # system overview
├── _patterns.md      # recurring patterns
├── _meta.yaml        # last compiled commit
├── modules/          # one article per module
├── concepts/         # cross-cutting concerns
├── decisions/        # why things are the way they are
├── learnings/        # bugs fixed, patterns discovered
└── queries/          # past Q&A, filed back

The first time you start a Claude Code session after init, the skill kicks in. The agent walks your codebase, reads the source files, and writes wiki articles. Module articles describe what each part of the code actually does. Not what it’s supposed to do, what it does. Key files, functions, data flow, connections to other modules.

Concept articles cut across modules. “How does error handling work across the system” or “how does data flow from request to response.” These are the questions that normally require reading eight files across four directories. The wiki answers them in one place.

Keeping it fresh

The wiki is only useful if it stays current. Every article has YAML frontmatter with a source_files field:

---
title: Authentication Module
type: module
source_files:
  - src/auth/middleware.py
  - src/auth/tokens.py
tags: [auth, middleware, jwt]
---

The CLI tracks which commit the wiki was last compiled against. When you start a new session, cw status diffs against that commit and cross-references changed files against every article’s source_files:

$ cw status
Changed since last compile (4964c23):
  M src/auth/middleware.py
  M src/auth/tokens.py

Stale articles:
  ! modules/auth.md

The agent sees this and knows exactly what to re-read and update. No guessing, no full recompile.

At session end, the agent writes learnings and decisions back into the wiki. Fixed a bug? That becomes learnings/auth-token-race-condition.md. Made a design decision? That’s decisions/switched-to-redis-sessions.md. Then it updates _meta.yaml with the current commit hash.

Next session picks up where this one left off.

The CLI

About 400 lines of Rust. Here are the commands:

cw init                # scaffold wiki for current repo
cw status              # what changed since last compile
cw path                # print wiki path
cw projects            # list all wikis
cw index               # rebuild _index.md from article frontmatter
cw meta update         # record current commit as compiled

cw setup claude-code   # install skill into Claude Code
cw setup codex         # install instructions for Codex
cw setup qmd           # register wiki as QMD search collection

The CLI doesn’t make any LLM calls. It handles the things agents are bad at: tracking git state, knowing which files changed, maintaining timestamps. The agent handles what it’s good at: reading code and writing about it.

Search with QMD

For larger wikis, QMD by Tobi Lutke adds proper search. It’s a local search engine for markdown with hybrid BM25 plus vector search plus a small reranker model. Running cw setup qmd registers your wiki as a searchable collection. The agent can then query the wiki through QMD’s MCP server during a session.

At the scale of most repos people actually work in, you probably don’t need it. A well organized wiki with an index file is enough for the LLM to navigate on its own. But when the wiki gets large, QMD keeps retrieval fast.

Viewing with Obsidian

All wiki articles live at ~/.codewiki/. Open that directory as an Obsidian vault and you get a browsable knowledge graph of all your projects. Articles use [[backlinks]] so modules connect to each other. The auth article links to [[database]] and [[api]]. You never have to write or edit these articles yourself. The agent maintains everything.

Why not RAG

Traditional RAG chunks your code, embeds it, retrieves fragments when you ask a question. You get decontextualized snippets and hope the LLM can stitch them together.

CodeWiki is different. The LLM reads the code and writes structured articles about it. The auth article already connects the middleware to the token service to the database layer. That connection doesn’t exist in any single source file. It exists in the compiled understanding.

Karpathy found the same thing with his research wiki. You don’t need vector search over raw data when you have a well organized collection of articles. The LLM reads the index, finds the relevant articles, reads those. Simple and it works.

Getting started

git clone https://github.com/mraza007/codewiki.git
cd codewiki
cargo install --path .

cd your-project
cw init
cw setup claude-code

Start a Claude Code session and the skill handles the rest. The project is MIT licensed and on GitHub.

Coding agents help with this. I can point Claude Code at an issue and let it work while I do something else. But that’s still one agent, one issue, one terminal. If I have 10 issues labeled “agent-ready,” I’m not babysitting 10 terminal tabs.

I wanted something that just watches for new issues and sends agents after them. Then OpenAI released their Symphony spec, an orchestrator pattern for their Codex agent. The architecture was solid: poll an issue tracker, dispatch agents into isolated workspaces, reconcile when issues close. But it was built around Codex and Linear, and I use Claude Code and GitHub Issues.

So I took the ideas I liked from Symphony and built my own. That’s Baton.

What it does

Baton is a Python daemon. You start it in your repo, it polls GitHub Issues matching your configured labels, creates an isolated git worktree per issue, and runs Claude Code CLI as a subprocess. When the agent finishes and opens a PR, Baton releases the claim and grabs the next issue.

One config file. One command. Go do something else.

WORKFLOW.md -> Orchestrator -> Worker (per issue)
                  |              |
                  |              +-- git worktree create
                  |              +-- hooks (before_run)
                  |              +-- claude -p "<prompt>"
                  |              +-- check issue state
                  |              +-- hooks (after_run)
                  |
                  +-- Poller (gh issue list)
                  +-- Dispatcher (concurrency control)
                  +-- Reconciler (stale run detection)

The name comes from relay races. You hand off the baton and the runner goes.

The config

Everything lives in WORKFLOW.md. YAML front matter for configuration, Jinja2 template below for the prompt. Baton reloads this file on every poll cycle, so you can change settings without restarting.

---
tracker:
  kind: github
  labels: ["agent"]
  exclude_labels: ["blocked"]

polling:
  interval_ms: 30000

agent:
  max_concurrent: 3
  max_turns: 5
  command: claude
  permission_mode: bypassPermissions

hooks:
  before_run: |
    git fetch origin main && git rebase origin/main
  timeout_ms: 60000
---

You are an autonomous software engineer working on issue #{{ issue.number " }}: {{ issue.title " }}.

{{ issue.body " }}

{% if attempt %}
This is continuation attempt {{ attempt " }}. Review what was done and continue.
{% endif %}

## Instructions

1. Understand the issue requirements
2. Write clean, well-tested code
3. Run existing tests to make sure nothing breaks
4. Commit your changes with a descriptive message
5. Push the branch and create a pull request linking to #{{ issue.number " }}

Labels filter which issues get picked up. max_concurrent controls parallel agents. max_turns is the retry limit per issue. Hooks run shell commands at different points. I use before_run to rebase on main so the agent starts from fresh code.

The prompt template gets issue.number, issue.title, issue.body, issue.labels, and attempt for retries. Standard Jinja2.

Why worktrees

Each issue gets its own worktree under .symphony/worktrees/, with a branch name slugified from the issue title: baton/fix-login-redirect-42.

I thought about Docker containers and temp directories but worktrees won out. They share the git object database so creating one is almost instant, unlike a full clone. They’re real checkouts, so linters and test runners and build scripts all work without any path hacking. And they’re isolated. If one agent trashes its branch, the others don’t care.

Why gh CLI instead of the GitHub API

Baton shells out to gh issue list and gh pr create instead of using PyGitHub or the REST API. Seems odd, but think about setup.

With the API, you need a personal access token. You need to configure it somewhere. You need to handle rate limits.

With gh, you authenticate once (gh auth login) and everything on your machine uses the same credentials. No token management in the orchestrator. The tradeoff is speed, but Baton polls every 30 seconds. The overhead of a subprocess call doesn’t matter at that pace.

The permission problem

This tripped me up. Claude Code has permission modes: default asks for everything, acceptEdits auto-approves file edits but prompts for shell commands, and bypassPermissions auto-approves everything.

I started with acceptEdits because it felt like the right balance. Let the agent write code freely, but make it ask before running commands. Problem: “ask” means a human clicking yes, and in an autonomous orchestrator there’s no human. The agent just blocks forever waiting for a prompt nobody will answer.

I wasted about 20 minutes watching it hang before I figured this out. For autonomous operation you need bypassPermissions, which maps to --dangerously-skip-permissions. The flag name is honest about the risk. I’m comfortable with it because the agents run in isolated worktrees on disposable branches, not in my main checkout.

Auto-releasing on PR creation

My first version had a dumb problem. The agent would finish its work, create a PR on turn 2 of 5, and Baton would keep scheduling continuation turns for the remaining 3. The slot was occupied but nobody was doing anything useful.

The fix: after each worker finishes, check if a PR exists for that issue’s branch. If yes, release the claim immediately and free up the slot. If not, schedule a short retry.

pr_exists = await self.tracker.check_pr_exists(issue.number)
if pr_exists:
    log.info(f"PR_READY #{issue.number} -- PR found, releasing claim")
    return "pr_created"
return "no_pr"

Small change, but it meant the orchestrator stopped wasting time on finished work.

Extensibility through skills and MCP servers

Baton itself is deliberately simple. It polls, dispatches, and manages worktrees. The interesting part is what you put in the prompt and what tools you give the agent.

Claude Code supports MCP servers, which means you can wire up external tools and the agent can use them during its run. Baton passes MCP server config through to each worker:

agent:
  mcp_servers:
    - name: playwright
      command: npx @playwright/mcp@latest

That means the agent has access to a headless browser while it works. It can open a page, click around, take screenshots, verify that the UI renders correctly. You don’t have to build that into Baton. You just declare which MCP servers you want and the agent figures out when to use them.

Same idea with CLI tools. If agent-browser is installed on the machine, you can tell the agent to use it in the prompt template. “Before creating a PR, open the app with agent-browser and verify the acceptance criteria.” The agent spins up a local server, opens the page, clicks buttons, fills inputs, takes snapshots. All from instructions in WORKFLOW.md, nothing hardcoded in the orchestrator.

Claude Code also has skills, which are reusable prompt fragments that teach the agent specific capabilities. If you have a code review skill or a testing skill installed, the agent can use them during its run. Baton’s config supports a skills list for this:

agent:
  skills:
    - code-reviewer
    - accessibility-checker

You can also override skills per issue by adding a ## Skills section to the issue body. If one issue needs Playwright but the others don’t, just add it to that issue.

The point is that Baton doesn’t need to know about browsers or test runners or linters. It just needs to dispatch agents with the right config. The prompt and the tools do the rest.

Putting it together: a todo app from scratch

To see all of this working end to end, I had Baton build a todo app. Fresh repo, no code. I created three GitHub issues labeled baton:

1. Create basic HTML structure
2. Add JavaScript for create/delete
3. Add localStorage persistence

The WORKFLOW.md prompt told the agent to use agent-browser for verification before opening PRs. I ran baton start and went to make coffee.

Baton picked up issue #1, created a worktree on baton/create-basic-todo-app-html-structure-1, and dispatched Claude Code. The agent wrote index.html, spun up a local server with npx serve, opened it with agent-browser, confirmed the layout rendered, then committed, pushed, and opened a PR. The PR description included what agent-browser found:

Opened http://localhost:3456 and confirmed the page renders correctly. Ran agent-browser snapshot -i confirming interactive elements: textbox and button.

I merged it. The issue auto-closed (the PR had Closes #1). Baton saw the issue was gone on the next poll, released the slot, and picked up issue #2. Same cycle. Then #3.

Three issues, three PRs, three merges. I didn’t write a line of the todo app. The agent-browser verification wasn’t built into Baton. It was just instructions in the prompt and a CLI tool on my machine.

Getting started

pip install -e .
cp WORKFLOW.md.example WORKFLOW.md
# Edit WORKFLOW.md: set your labels, tweak the prompt
baton start

You need Python 3.11+, Claude Code CLI (claude), GitHub CLI (gh) authenticated, and Git.

The code is at github.com/mraza007/baton. MIT licensed. About 10 Python modules, no external services, no databases. State lives in memory with JSON persistence for the status command.

What I want to add next

• A proper TUI instead of baton status reading a JSON file
• Issue dependency ordering so issue 3 waits for issue 2 if it needs to
• Cost tracking per issue, so I can see what automating the backlog actually costs in tokens
• More trackers besides GitHub Issues (Linear, Jira, GitLab)

If you’ve got a repo with a pile of issues sitting there, try pointing Baton at it. Start with one label and max_concurrent: 1. See what it does. The setup takes about five minutes and the worst case is you get a bad PR that you close. The code is MIT licensed, the whole thing is ten files, and there’s nothing weird in it. Fork it, break it, rip out the parts you don’t like.

If you try it, I want to hear what breaks.

I write a newsletter called Dev Console where I cover what’s actually happening in AI, minus the hype. New tools, real use cases, stuff I’m building. If this post was interesting, you’ll probably like it.

Harness engineering. It’s the practice of building everything around the LLM — the execution environment, tool definitions, safety boundaries, observability, and lifecycle management. The stuff that turns a chatbot into a production system.

If you work in DevOps, you’ve been doing this for years. You just called it something else.

Why Harnesses Matter More Than Models

Pick any AI agent demo. Strip out the model. What’s left?

A container or sandbox. A set of callable tools. A loop that reads output and decides what happens next. Logging. Timeouts. Cleanup.

That’s the harness. And it’s where agents succeed or fail. A great model in a bad harness hallucinates, loops forever, leaks secrets, or silently does nothing useful. A decent model in a good harness stays bounded, recovers from errors, and produces auditable results.

DevOps engineers already think this way. You don’t just pick a good application — you build the infrastructure that makes it reliable. Same thing here.

The Five Parts of a Harness

Here’s how I break down harness engineering into components. Each one maps directly to something you’ve built before.

1. Execution environment. Where does the agent run? A container, a VM, a temporary directory, a git worktree. You need isolation so the agent can’t corrupt shared state. You need reproducibility so runs are consistent. This is the same problem as CI job runners. Docker, Firecracker, nsjail — pick your isolation boundary.

2. Tool definitions. Tools are the agent’s API surface. Read a file. Run a command. Query a database. Call an endpoint. Each tool needs input validation, output formatting, error handling, and permission scoping. Think of it like designing an API — you wouldn’t expose raw database access through a REST endpoint. Don’t give an agent raw shell access either. The tool layer is your contract.

3. Control loop. Observe, decide, execute, verify. The loop is what makes an agent an agent instead of a one-shot prompt. Your job as a harness engineer is to decide: how many iterations? What’s the timeout per step? What happens when a tool call fails? When does the loop escalate to a human? This is the same logic you put in health check loops and deployment rollback controllers.

4. Guardrails. Cost caps. Token limits. Command allowlists. File path restrictions. Rate limiting on external calls. Without guardrails, an agent can burn through your API budget in minutes or write to paths it shouldn’t touch. Every guardrail is a policy decision — same as IAM policies, network rules, and resource quotas you already manage.

5. Observability. If you can’t see what the agent did, you can’t debug it, audit it, or trust it. Log every tool call, every LLM response, every decision point. Capture diffs, timing, token usage, and cost. This is no different from structured logging in any production system. The difference is that agent traces are longer and less predictable than HTTP request traces, so you need good tooling to navigate them.

Where DevOps Context Overlaps

Here’s where your existing skills plug in directly.

Infrastructure as code. Agent harnesses should be declarative and version-controlled. The tool definitions, policies, and environment specs should live in config files, not hardcoded in application logic. When you change a tool’s behavior, that change should be reviewable in a PR.

Pipeline orchestration. Multi-agent systems look a lot like multi-stage pipelines. One agent does research, passes context to a planning agent, which passes a plan to an implementation agent. You’re managing handoffs, shared artifacts, and failure propagation — the same coordination problem as CI/CD stages.

Incident response. When an agent goes wrong, you need the same muscle memory. Check the logs. Find the failing step. Understand the input that caused it. Roll back if needed. The debugging workflow is identical.

Security boundaries. Least privilege applies to agents just like it applies to services. What tools can this agent access? What files can it read? Can it make network calls? Can it spend money? Every agent needs a security boundary, and DevOps engineers already think in terms of boundaries.

Getting Started

If you want to start building harnesses, you don’t need a new framework. Start with what you have.

Take a simple task — say, analyzing a failed CI build. Write a script that collects the logs, sends them to an LLM with a prompt, parses the response, and posts a summary to Slack. That’s a harness. A minimal one, but it has all the components: environment setup, tool use (log collection, Slack posting), a control flow, and output handling.

Then add complexity. Let the LLM decide which logs to fetch. Add a retry loop. Add a cost cap. Add structured logging. Each addition is a harness engineering decision.

You don’t need to learn ML. You don’t need to fine-tune models. You need to build the infrastructure that makes models useful — and that’s the job you already do.

Harness engineering isn’t a new discipline. It’s DevOps applied to a new kind of workload. The sooner you see it that way, the faster you’ll build agents that actually work in production.

You start a new session and it re-discovers the same patterns. Repeats the same mistakes. Asks the same questions. It’s like working with a brilliant colleague who gets amnesia every night.

I got tired of this. So I built EchoVault — a local memory system that gives coding agents persistent memory across sessions. No cloud. No API keys. No cost. Just a SQLite database and some Markdown files on your machine.

The Problem Is Real

I use coding agents daily across multiple client projects. Claude Code, Cursor, Codex — I switch between them depending on the task. Every time I start a session, I’m repeating context that the agent should already know.

“We chose FastAPI over Flask because of async support.” “The deploy script needs –no-cache or the CSS breaks.” “Don’t touch the legacy auth module — it’s being replaced next sprint.”

I was copy-pasting this stuff into every session. That’s not how tools should work.

I tried existing solutions. Supermemory announced their MCP and I was tempted, but it saves everything in the cloud. I work with multiple companies as a consultant — I don’t want codebase decisions stored on someone else’s servers. Claude Mem was the first tool I tried, but it was eating too much memory in my sessions and became a bottleneck when running multiple agents at the same time.

So I built my own.

How EchoVault Works

EchoVault runs as an MCP server. When your agent starts a session, it has three tools available:

• memory_context — load prior decisions, bugs, and context for the current project
• memory_search — find specific memories by keyword or semantic similarity
• memory_save — persist a decision, bug fix, pattern, or learning

The agent calls these tools like it calls any other tool. No hooks. No shell scripts. No prompt injection. The MCP protocol handles everything.

Here’s what happens in practice:

Session start. The agent sees memory_context in its available tools. The tool description says “You MUST call this at session start.” The agent calls it and gets back a list of prior memories for the project. Now it knows what happened yesterday.

During work. You ask about authentication. The agent calls memory_search with “authentication” and gets back the decision to use JWT, the bug with token refresh, and the migration plan. It has context before writing a single line of code.

Session end. The agent just fixed a tricky race condition. The tool description says “You MUST call memory_save before ending any session where you made changes.” It saves the root cause, the fix, and what to watch for.

Next session, that knowledge is there. Every session builds on the last one.

The Architecture

I kept it simple. The whole system is four things:

~/.memory/
├── vault/                    # Obsidian-compatible Markdown
│   └── my-project/
│       └── 2026-02-01-session.md
├── index.db                  # SQLite: FTS5 + sqlite-vec
└── config.yaml               # Optional embedding config

Markdown vault. Every memory gets written to a session file — one file per day per project. These are valid Markdown with YAML frontmatter. You can point Obsidian at ~/.memory/vault/ and browse your agent’s memory visually. You can read them in any editor. They’re not locked in a proprietary format.

SQLite index. This is where search happens. FTS5 handles keyword search out of the box — no configuration needed. If you want semantic search (where “authentication” matches a memory titled “JWT token setup”), add an embedding provider. I use Ollama with nomic-embed-text locally. You can also use OpenAI or OpenRouter if you prefer cloud.

MCP server. The agent talks to EchoVault through the Model Context Protocol. Three tools, stdio transport, nothing fancy. The server starts when the agent needs it and stops when the session ends. Zero idle cost.

Secret redaction. Three layers. Explicit <redacted> tags for things you mark yourself. Pattern detection that catches API keys, passwords, and credentials automatically. And .memoryignore rules for custom patterns. Nothing sensitive hits disk.

Making Agents Actually Save

Here’s the thing about MCP tools — the agent can call them, but will it? Retrieval works well because agents tend to grab context at the start. Saving is the hard part. The agent finishes its work and moves on. It doesn’t naturally think “I should save what I learned.”

The trick is the tool descriptions. When you register an MCP tool, you include a description. Agents read these descriptions and treat them as instructions. So instead of:

"Save a memory for future sessions. Call this when you make decisions."

I wrote:

"Save a memory for future sessions. You MUST call this before ending
any session where you made changes, fixed bugs, made decisions, or
learned something. This is not optional — failing to save means the
next session starts from zero."

That “MUST” language makes a real difference. It’s not 100% reliable — nothing with LLMs is — but agents follow strong tool descriptions much more consistently than passive ones.

Cross-Agent Memory

One of the things I wanted was a single vault for all my agents. A memory saved by Claude Code should be searchable from Cursor or Codex. They’re all working on the same codebase. Why should they have separate memories?

EchoVault stores everything in one place. The MCP server is the same regardless of which agent connects to it. Setup is one command per agent:

memory setup claude-code   # writes ~/.claude.json
memory setup cursor        # writes .cursor/mcp.json
memory setup codex         # writes .codex/config.toml + AGENTS.md
memory setup opencode      # writes opencode.json

Each agent has its own config format and conventions. Claude Code uses JSON with mcpServers. Cursor uses the same schema but different file paths. Codex uses TOML with [mcp_servers]. OpenCode uses JSON with a mcp key and a different command format (command as an array instead of separate command + args).

I wrote shared helpers so each agent’s setup is just a thin wrapper around _install_mcp_servers() or _install_toml_mcp(). Adding a new agent takes maybe 20 lines of code.

What Gets Saved

Not everything should be a memory. Trivial changes don’t need to be persisted. Information that’s obvious from reading the code doesn’t need a memory. The goal is to capture what a future agent wouldn’t know from just looking at the codebase.

Good memories:

• Decisions. “Chose JWT over sessions because the API needs to be stateless.” A future agent reading the code sees JWT but doesn’t know why.
• Bugs. “The ORM lazy-loads relationships by default, causing N+1 queries in the user list endpoint. Fixed by adding .options(joinedload(...)). Root cause: SQLAlchemy default behavior.” A future agent won’t hit the same bug.
• Patterns. “All API endpoints follow the pattern: validate input, check permissions, execute, return response. Don’t add business logic in the route handler.” A future agent follows the existing patterns instead of inventing new ones.
• Context. “The legacy auth module is being replaced. Don’t modify it — changes go into the new auth service at src/auth/v2/.” A future agent doesn’t waste time on dead code.

Each memory has a title, a “what happened” summary, optional “why” and “impact” fields, tags, and a category. Search returns compact ~50-token summaries. Full details are fetched on demand so context windows don’t get bloated.

The Technical Bits

A few implementation details that might be useful if you’re building something similar.

FTS5 for keyword search. SQLite’s FTS5 extension is fast and works with zero configuration. No external service needed. It handles stemming, phrase matching, and ranking. For most use cases, this is all you need.

sqlite-vec for semantic search. When you want “authentication” to match “JWT token rotation”, you need vectors. I use sqlite-vec to store embeddings right in the same SQLite database. No vector database needed. Embedding providers are pluggable — Ollama for local, OpenAI or OpenRouter for cloud.

Hybrid search. The search pipeline runs FTS5 first (fast, precise), then semantic search (slower, fuzzy), and merges the results. This gives you the best of both worlds — exact keyword matches and semantic similarity.

TOML parsing with fallbacks. Codex writes some non-standard TOML — unquoted filesystem paths as table keys, dotted version strings as key names. Standard tomllib chokes on these. I added a fallback that appends the MCP section directly via string operations when parsing fails. It’s not pretty but it handles real-world config files.

Symlink handling. Some agents create symlinks in their skill directories. shutil.rmtree() crashes on symlinks. Small thing but it bit me in production.

Setting It Up

pip install git+https://github.com/mraza007/echovault.git
memory init
memory setup claude-code

That’s it. Three commands. The agent has memory now.

If you want semantic search, configure an embedding provider:

memory config init
# Edit ~/.memory/config.yaml to set your provider
memory reindex

For fully local operation with no external API calls, use Ollama:

embedding:
  provider: ollama
  model: nomic-embed-text

What I’ve Learned

Building this taught me a few things about agent tooling.

Tool descriptions are instructions. Agents read them and follow them. Strong, directive language in tool descriptions is more effective than passive documentation. “You MUST” works better than “You can.”

Local-first matters. Not because of ideology, but because of practical constraints. Consultants work with multiple clients. Sensitive decisions shouldn’t leave the machine. And when your internet goes out, local tools still work.

MCP is the right abstraction. Instead of writing agent-specific hooks, skills, and config formats, I write one MCP server and each agent connects to it. When a new agent comes along, I add a setup function for its config format. The memory logic doesn’t change.

Simple storage wins. Markdown files you can read in any editor. SQLite you can query with any tool. No custom binary formats. No daemon to keep running. The system is completely inspectable and debuggable.

The code is at github.com/mraza007/echovault. It’s MIT licensed. If you’re tired of your agents forgetting everything, give it a shot.

Now swap out the hardcoded logic for an LLM. That’s it. That’s an AI agent in simpler terms. The fancy demos want you to think it’s magic. Some brand new thing you need to learn from scratch. It’s not. When you take away the hype, an agent is just a controlled automation loop. The LLM handles the reasoning and everything else is infrastructure you’ve built a hundred times.

Here’s what matters, the agent itself isn’t the hard part but The harness is, the execution environment, tooling, guardrails, and observability. It’s all the important stuff that makes automation work in production.

DevOps engineers have been building harnesses forever. CI runners. Deployment pipelines. Infrastructure automation. The patterns are the same. The skills transfer directly.

So if you’re wondering whether AI agents are worth learning, here’s the short answer. You’re already halfway there.

What an Agent Actually Looks Like

Let’s forget the marketing hype around AI agents and understand from a DevOps engineer’s point of view, what an agent actually looks like. An AI agent has six parts.

1. An LLM: Now LLM is the most important part of an agent as this acts as a brain. It reads context and decides what to do next. It doesn’t touch anything directly.

2. A workspace: Think of it as a sandboxed environment. A cloned repo. A container. A temp directory. Same as any CI job.

3. A set of tools: These are the actions it can request. Read a file. Run a command. Call an API. Query logs. The agent doesn’t run these itself. It asks for them.

4. A control loop: This is the core pattern. Observe the current state. Decide an action. Execute it. Check the result. Keep going until you’re done.

5. Policies and limits: Timeouts. Permission boundaries. Rate limits. Cost caps. Without these, agents can spin forever or do things they shouldn’t.

6. A termination condition: The agent needs to know when to stop. Task complete. Error threshold hit. Human review needed. Something has to end the loop.

Now none of this is new as you’ve built systems with all these components. The only difference is the LLM sitting in the decision seat.

The Harness Does the Heavy Lifting

Everyone focuses on the LLM. They miss the important part. The harness is what makes an agent actually work.

The harness is everything around the model. It spins up the environment. Exposes tools. Executes commands on the agent’s behalf. Captures logs and diffs. Enforces limits. Decides when the loop should stop.

Sound familiar? It should. This is what CI runners do.

GitHub Actions. GitLab runners. Jenkins agents. They all follow the same pattern. Spin up an isolated environment. Run steps. Capture output. Handle success and failure. Clean up.

An agent harness does the exact same thing. The only twist is the steps aren’t hardcoded in YAML. They come from the LLM at runtime.

This is why DevOps engineers are perfect for this work. You already think about isolation, execution, logging, and cleanup. You already build systems that run untrusted code safely. Agent harnesses are the same problem with a new input source.

Tool Use Is the Safety Mechanism

Agents don’t touch systems directly. This matters. The LLM never runs a command itself. Never writes a file itself. It requests actions through tools.

The harness gets the request. Validates it. Executes it in a controlled way. Returns a structured result.

This is how you keep agents safe.

Say the agent wants to run a shell command. The harness can check it against an allowlist. Run it in a sandbox. Set a timeout. Capture stderr. The agent never gets raw shell access.

Same thing for file operations. The agent requests a file write. The harness checks the path. Validates the content. Writes the file and returns confirmation.

You control what tools exist. You control how they behave. You control what the agent can even ask for.

This is the same idea behind least privilege. The agent only gets access to what it needs. The harness enforces the boundary.

The Control Loop in Practice

The core of any agent is the control loop. It looks like this.

1. Observe: The agent reads the current state. Test output. Log files. Diffs. Error messages. Whatever context it needs.

2. Decide: The LLM looks at the state and picks an action. Run another test. Edit a file. Ask for more information. Give up.

3. Execute: The harness runs the requested action and returns the result.

4. Verify: The agent checks if the action worked. Did the test pass? Did the error go away? Is the task done?

5. Repeat: If the task isn’t complete, go back to observe.

This loop keeps running until a termination condition hits—success, failure, timeout, max iterations, or human intervention.

You’ve seen this before: build, test, fix, rebuild. CI pipelines do this, deployment rollbacks do this, and health check loops do this.

Agents just make the “decide” step dynamic instead of scripted, and here’s where they actually help in DevOps work.

CI failure analysis. When a test fails, the agent reads the logs, checks the diff, identifies the cause, and suggests a fix—maybe even applying it and rerunning the test.

Terraform drift detection. The agent compares actual state to declared state, flags the drift, and proposes a remediation plan while a human approves before anything changes.

Kubernetes manifest review. The agent checks YAML against best practices (missing resource limits, no liveness probes, exposed secrets) catching the stuff humans miss in review.

Cost anomaly investigation. When spending spikes, the agent queries cost explorer, correlates with recent deployments, and surfaces the likely cause, saving an hour of digging.

Incident log triage. Faced with pages of logs, the agent reads them, extracts the relevant lines, and summarizes what went wrong (not replacing the engineer, but getting them to the answer faster).

Notice the pattern: the agent assists and handles the tedious parts while the human stays in control of decisions that matter.

AI agents sound complicated with their new frameworks, new terminology, and new paradigms.

But look past the hype and you’ll see something familiar.

An agent is an automation loop where the LLM picks the next step, the harness executes it safely, tools provide controlled access to systems, and policies keep things bounded.

This is CI/CD architecture, infrastructure thinking, the stuff you already do.

When you read about agent frameworks or watch demos of coding assistants, you now have a lens to see the harness underneath, spot the control loop, and ask the right questions: what tools does it expose, what limits exist, and how does it handle failure?

You don’t need to become an ML engineer to understand agents—you just need to recognize the infrastructure patterns you’ve been using all along.

The LLM is the new part. Everything else is your domain.

After one week of systematic analysis, I cut it to $2,500/month — a 50% reduction, saving them $30,000 annually. Here’s exactly how I did it, with the scripts you can use.

The Discovery Phase: How I Found the Problems

Before touching anything, I needed to understand the infrastructure. Here’s my systematic approach:

Step 1: Pull Cost Data by Service

First, I analyzed their AWS Cost Explorer data to understand where money was going:

aws ce get-cost-and-usage \
  --time-period Start=2024-11-01,End=2024-11-30 \
  --granularity MONTHLY \
  --metrics "BlendedCost" \
  --group-by Type=DIMENSION,Key=SERVICE

This gave me the high-level breakdown. But I needed more detail.

Step 2: Build a Resource Inventory

I wrote a Python script to scan all resources across regions and identify optimization opportunities:

import boto3

def scan_ebs_volumes():
    """Find GP2 volumes that should be GP3 and unattached volumes."""
    ec2 = boto3.client('ec2')
    volumes = ec2.describe_volumes()['Volumes']

    gp2_volumes = []
    unattached = []

    for vol in volumes:
        if vol['VolumeType'] == 'gp2':
            gp2_volumes.append({
                'VolumeId': vol['VolumeId'],
                'Size': vol['Size'],
                'State': vol['State'],
                'MonthlyCost': vol['Size'] * 0.10,  # GP2 pricing
                'GP3Cost': vol['Size'] * 0.08,      # GP3 pricing
                'Savings': vol['Size'] * 0.02
            })

        if vol['State'] == 'available':  # Not attached
            unattached.append(vol)

    return gp2_volumes, unattached

I built similar functions for:

• RDS instances (storage type, utilization, Multi-AZ necessity)
• EC2 instances (generation, Reserved Instance coverage)
• Elastic IPs (attached vs idle)
• EBS snapshots (age, associated volumes)
• S3 buckets (storage class, lifecycle policies)

Step 3: Analyze CloudWatch Metrics for Utilization

This is critical. Before recommending any right-sizing, I needed data:

def get_instance_utilization(instance_id, days=30):
    """Get average CPU utilization over the past N days."""
    cloudwatch = boto3.client('cloudwatch')

    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/EC2',
        MetricName='CPUUtilization',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        StartTime=datetime.now() - timedelta(days=days),
        EndTime=datetime.now(),
        Period=86400,  # Daily
        Statistics=['Average', 'Maximum']
    )

    return response['Datapoints']

The results were eye-opening:

• Two t3.xlarge instances averaging 12% CPU
• RDS storage at 95% free space
• Multiple log groups with no retention policy (storing terabytes)

Step 4: Map Dependencies Before Cutting

Before deleting anything, I mapped what depended on what:

• Which services used which Elastic IPs?
• Which applications wrote to which log groups?
• Which backups were actually needed for compliance?

This prevented the classic mistake of breaking production while optimizing costs.

The Starting Point

After the discovery phase, here’s what I was working with:

The distribution told me a lot. EC2-related costs (compute + EBS + networking) made up over 38% of the bill. That’s where I started.

Phase 1: Quick Wins (Implemented Same Day)

Release Idle Elastic IPs — Saved $50/month

My inventory script flagged 5 Elastic IPs with no association:

aws ec2 describe-addresses --query 'Addresses[?AssociationId==null]'

Someone had provisioned them for test environments that were deleted months ago. Classic ghost infrastructure.

Time to fix: 5 minutes.

Migrate EBS GP2 to GP3 — Saved $125/month

The script found 6,000+ GB across multiple EBS volumes still on GP2. GP3 costs 20% less and provides better baseline performance (3,000 IOPS vs GP2’s variable IOPS based on size).

aws ec2 modify-volume --volume-id vol-xxx --volume-type gp3

No downtime. Just a CLI command per volume.

Time to fix: 30 minutes for all volumes.

Set CloudWatch Log Retention — Saved $100/month

My scan found 20+ log groups with no retention policy — storing logs forever:

aws logs describe-log-groups --query 'logGroups[?retentionInDays==null].logGroupName'

Set production to 90 days, staging to 30 days.

Time to fix: 20 minutes.

Phase 2: The Big Discoveries

AWS Backup Running 24x More Often Than Needed — Saved $400/month

This was the most surprising find. When I pulled the backup plan configuration:

aws backup get-backup-plan --backup-plan-id xxx

I saw: hourly backups. 24 backups per day. For every resource.

The backup storage had grown to $500/month — 10% of their total bill.

I reviewed their recovery requirements (they only needed daily backups with 14-day retention) and reconfigured:

{
  "ScheduleExpression": "cron(0 5 * * ? *)",
  "StartWindowMinutes": 60,
  "CompletionWindowMinutes": 120,
  "Lifecycle": {
    "DeleteAfterDays": 14
  }
}

Time to fix: 1 hour (including testing).

CloudWatch Metric Streams to Nowhere — Saved $400/month

My CloudWatch cost breakdown showed $400/month on “Metric Streams” — 100+ million metric updates going somewhere.

aws cloudwatch list-metric-streams

Found a stream configured to send data to a third-party monitoring tool. When I asked about it, nobody on the team knew it existed. The integration had been set up by a previous contractor and was never used.

This is a perfect example of ghost infrastructure that accumulates over time.

RDS Over-Provisioned by 95%

My RDS analysis showed all instances had massive storage allocated. The CloudWatch metrics told the real story:

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name FreeStorageSpace \
  --dimensions Name=DBInstanceIdentifier,Value=production-db \
  --start-time 2024-11-01T00:00:00Z \
  --end-time 2024-11-30T23:59:59Z \
  --period 86400 \
  --statistics Average

Result: 95% free space across all databases.

RDS storage can only be increased, not decreased. But I migrated all instances from GP2 to GP3 storage — same price, better performance.

For the next database refresh, I recommended right-sized storage instead of the default massive allocations.

Saved: $150/month

Phase 3: Infrastructure Improvements

NAT Gateway Consolidation — Saved $125/month

My VPC analysis showed NAT Gateways in every AZ across multiple regions costing $500/month combined. After reviewing their architecture and traffic patterns, they only needed half of them.

ECS Task Right-Sizing — Saved $250/month

The ECS service scan found:

• A staging service constantly failing health checks and restarting (consuming resources 24/7 while accomplishing nothing)
• Legacy services still running in production that nobody was using

These issues relate directly to the ECS architectural decisions that often waste weeks of engineering time. Plus, enabled Fargate Spot for fault-tolerant workloads (70% savings on those tasks).

S3 Lifecycle Policies — Saved $150/month

My S3 bucket analysis showed backup buckets had grown to 10+ TB with no lifecycle policy. Old backups were stored in Standard tier forever.

Added a policy to transition to Glacier after 90 days:

{
  "Rules": [
    {
      "ID": "archive-old-backups",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Reserved Instances for Stable Workloads — Saved $500/month

My EC2 coverage analysis showed zero Reserved Instance coverage on instances running 24/7.

I helped them purchase Compute Savings Plans covering their steady-state workloads. Immediate 30-40% savings on compute.

EC2 Instance Right-Sizing — Saved $250/month

The utilization data was clear: multiple instances running at 10-15% CPU.

Downsized t3.xlarge instances to t3.large where utilization data supported it. Same workload, half the cost.

The Results

From $5,000/month to $2,500/month — exactly 50% reduction.

Over a year, that’s $30,000 back in their pocket.

The Methodology

Here’s the systematic approach I use for every cost optimization engagement:

1. Get the Data First

Before making any changes, I pull:

• AWS Cost Explorer data (by service, by tag, over time)
• CloudWatch metrics for utilization
• Resource inventory across all regions

2. Find the Ghosts

“Ghost infrastructure” costs more than you think:

• Unused Elastic IPs
• Detached EBS volumes
• Empty S3 buckets accumulating requests
• Log groups with infinite retention
• Metric Streams nobody monitors
• Test environments that outlived their purpose

3. Right-Size Ruthlessly

Check actual utilization before committing to Reserved Instances:

# EC2 CPU utilization over 30 days
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-xxx \
  --start-time $(date -d "30 days ago" +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date +%Y-%m-%dT%H:%M:%S) \
  --period 3600 \
  --statistics Average

If a t3.xlarge averages 15% CPU, you’re paying for 85% idle capacity.

4. Modernize Storage

GP2 → GP3 is almost always worth it:

• 20% cheaper at baseline
• Better performance (3,000 IOPS baseline)
• Zero downtime migration

5. Review Backup Policies

Backups grow silently. Questions to ask:

• How often do you actually need backups?
• How long do you really need to keep them?
• Are you backing up dev/test environments at production frequency?

What This Looks Like Over Time

The best part? None of these changes affected performance or reliability. Most improved it.

Common Patterns I See

After doing this for multiple clients, patterns emerge:

1. Metric Streams nobody monitors — $100-400/month just disappearing
2. Hourly backups for daily restore needs — 24x the storage cost
3. GP2 volumes from years ago — never migrated to GP3
4. Multi-AZ staging databases — paying for HA nobody needs
5. NAT Gateways in every AZ — when one or two would suffice
6. Logs kept forever — “just in case”
7. No Reserved Instances — paying full on-demand for 24/7 workloads
8. Over-provisioned everything — “it might need it someday”

Need Help With Your AWS Bill?

I do AWS cost optimization as part of my DevOps consulting practice. If your AWS bill feels too high or you just want a second pair of eyes on your infrastructure, let’s talk.

Book a free 30-minute call — I’ll review your current setup and tell you where I see opportunities.

Have questions about any of these optimizations? Drop a comment below or reach out on Twitter/X.

The problem isn’t that ECS is hard. The problem is that teams treat infrastructure decisions like they’re permanent. They’re not.

Last year I worked with a startup that spent 5 weeks evaluating container orchestration options. Five weeks. They had a working app. They had paying customers waiting. But the engineering team was stuck in an endless loop of “what if we need to scale?” and “shouldn’t we future-proof this?”

They launched on Fargate. It took 3 days once they stopped debating.

Here are the 5 decisions that waste the most time and what I tell every client to pick.

Fargate vs EC2: Just Use Fargate

This one wastes more time than all the others combined.

I get it. EC2 looks cheaper on paper. You can run the numbers, build a spreadsheet, show that at 50 containers you’ll save $400/month with EC2. The finance person gets excited. Someone mentions spot instances. Now you’re three meetings deep into capacity planning for traffic you don’t have yet.

Here’s what actually happens with EC2: you spend a week figuring out instance types, another week on auto-scaling groups, then you hit some weird issue where your containers won’t place because the bin-packing algorithm can’t find space, and suddenly your “cheaper” option has eaten two sprints of engineering time.

Fargate just works. You tell it how much CPU and memory you need, and it runs your container. No instances to manage, no patching, no capacity planning.

“But it’s more expensive!”

Sure. Maybe 20-30% more at scale. But you’re not at scale. You’re trying to ship. And even if Fargate costs you an extra $200/month right now, that’s nothing compared to the $30k+ in engineering salaries you’re burning while debating this.

Python apps especially benefit from Fargate. Your Django app with Celery workers is memory-heavy and I/O bound. You’re not doing CPU-intensive work. Fargate lets you right-size memory without playing Tetris with EC2 instance types.

Pick Fargate. When you’re running 200 containers 24/7 and have real cost data, revisit. Until then, move on.

ECS Service Discovery: Use an Internal ALB

When your services need to talk to each other, AWS gives you three options: Cloud Map, internal ALB, or Service Connect. I’ve seen teams spend weeks evaluating all three, setting up proof-of-concepts, reading whitepapers.

Just use an internal ALB.

I know, it’s not great. It’s a load balancer. It’s been around forever. But that’s exactly why you should use it:

• It gives you a stable DNS name your services can call
• Health checks work out of the box
• You get access logs for debugging
• Every developer on your team already understands HTTP

Your FastAPI service calls http://api-internal.yourdomain.local/users and it just works. No service mesh. No Envoy sidecars. No DNS caching gotchas.

Cloud Map is fine, but I’ve debugged too many issues where services couldn’t find each other because of DNS TTL problems. Service Connect is powerful, but now you’re operating a service mesh. Do you really want to be debugging Envoy proxy configuration when your actual problem is a database query?

The internal ALB is boring. Boring is good. Boring means you’re debugging your application code instead of your infrastructure.

CI/CD for ECS: Use GitHub Actions

I’m gonna be honest here: if your code is on GitHub, use GitHub Actions. Don’t overthink this.

“But CodePipeline is AWS-native!”

Yes, and it requires you to set up a pipeline with Source, Build, and Deploy stages, configure IAM roles for each stage, create buildspec files, and wire everything together. It’s more YAML for the same result.

“But Jenkins gives us more control!”

It’s 2025. Please don’t set up a Jenkins server. You’ll spend more time maintaining Jenkins than deploying your app.

GitHub Actions has an official AWS action that handles ECS deployments:

- name: Deploy to ECS
  uses: aws-actions/amazon-ecs-deploy-task-definition@v1
  with:
    task-definition: task-definition.json
    service: my-service
    cluster: my-cluster
    wait-for-service-stability: true

That’s the whole thing. It registers your task definition, updates the service, and waits for the deployment to stabilize. AWS maintains it. It works.

Your deployment workflow lives in your repo, your team already knows GitHub Actions from running tests, and you’re not managing another piece of infrastructure. If you want to understand how these pipeline runners actually work internally, I wrote a deep dive on building a CI/CD pipeline runner from scratch in Python.

If you’re on GitLab, use GitLab CI. If you’re on Bitbucket, use Bitbucket Pipelines. The point is: use whatever’s already integrated with your code. Don’t add complexity.

ECS Secrets Management: Use SSM Parameter Store

Where do you store your database passwords and API keys?

Not in your task definition. I’ve seen that. Please don’t.

The two real options are SSM Parameter Store and Secrets Manager. Teams debate this endlessly because Secrets Manager has automatic rotation and sounds more “enterprise.”

Here’s the thing: SSM Parameter Store is free, integrates natively with ECS, and handles 99% of use cases.

{
  "secrets": [
    {
      "name": "DATABASE_URL",
      "valueFrom": "arn:aws:ssm:us-east-1:123456789:parameter/myapp/database_url"
    }
  ]
}

Your Python app reads os.environ['DATABASE_URL'] like it does locally. No SDK, no code changes.

Secrets Manager costs $0.40 per secret per month and is worth it if you need automatic rotation for RDS credentials. But you probably don’t need that on day one. Start with SSM, migrate specific secrets to Secrets Manager later if you need rotation.

And please, don’t set up HashiCorp Vault unless you have compliance requirements that specifically mandate it. You’re now operating a distributed system just to store passwords. That’s not simplifying your life.

ECS Logging and Monitoring: Use CloudWatch

Every team wants to evaluate Datadog, New Relic, Honeycomb, and then maybe self-host Prometheus and Grafana “for cost savings.”

Stop. Use CloudWatch.

Add this to your task definition:

{
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/my-service",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs"
    }
  }
}

Done. Your container logs go to CloudWatch. You can query them with Log Insights. Enable Container Insights and you get CPU/memory metrics. Set up a few alarms. You now have better observability than 80% of startups.

Datadog is genuinely good software. I like it. But it costs $15+ per host per month from day one, and you need to manage another vendor relationship. You can add it later when you actually need distributed tracing or APM.

Self-hosted observability is a trap. I’ve seen teams spend months building ELK stacks and Prometheus clusters. That’s infrastructure work that doesn’t ship features. Unless you have a dedicated platform team, don’t volunteer for this.

The Actual Pattern Here

Look at what I recommended:

• Fargate over EC2
• Internal ALB over Cloud Map or Service Connect
• GitHub Actions over CodePipeline or Jenkins
• SSM over Secrets Manager or Vault
• CloudWatch over Datadog or self-hosted

Every single choice optimizes for the same thing: less stuff to manage.

Yes, some of these cost slightly more money. Yes, some of them are less flexible. But they all share one property: they let you ship faster and debug easier.

And here’s what nobody puts in their architecture decision records: all of these choices are reversible.

• Fargate to EC2? Task definitions work on both.
• ALB to Service Connect? Just DNS changes.
• SSM to Secrets Manager? Same integration pattern.
• CloudWatch to Datadog? Add the agent, keep CloudWatch as backup.

The “wrong” choice costs you maybe a few hundred dollars a month in inefficiency. The debate about the “right” choice costs you weeks of engineering time.

What I’ve Actually Seen Happen

Teams that follow this advice ship in about a week:

• Day 1-2: Fargate cluster up, first service running
• Day 3: ALB routing traffic, services talking to each other
• Day 4: GitHub Actions deploying on push to main
• Day 5: Secrets in SSM, logs in CloudWatch, basic alarms set up

Week 2: Building features.

Teams that “do it right” are still having meetings about networking topology in week 6.

I’ve watched startups run out of runway while their infrastructure was still “almost ready.” I’ve seen senior engineers burn out on DevOps work instead of building the product that got them excited in the first place.

Your Python app on Fargate with CloudWatch logs isn’t going to fall over at 1,000 users. Probably not at 10,000. By the time scale is actually a problem, you’ll have the traffic data and revenue to solve it properly.

Ship first. Optimize later.

If you found this helpful, share it on X and tag me @muhammad_o7 - I’d love to hear about your ECS deployment experiences. You can also connect with me on LinkedIn.

Need Help? I’m available for AWS and DevOps consulting. If you’re stuck in ECS decision paralysis or need help getting to production faster, reach out via email or DM me on X/Twitter.

But here’s the problem: that pipeline runner can only do exactly what you tell it to do.

It’s 2 AM. Your deployment pipeline fails. The error message is cryptic: Error: Connection refused on port 5432. Your traditional CI/CD pipeline stops dead. It sends an alert. You wake up, check the logs, realize the database connection pool was exhausted, restart the service, and go back to bed frustrated.

What if your pipeline could investigate the failure itself?

What if, instead of just stopping and alerting you, it could:

• Analyze the error logs
• Check recent code changes
• Search for similar issues in your repository
• Identify that this same error happened two weeks ago when someone forgot to increase the connection pool
• Post a detailed root cause analysis to Slack with a suggested fix

That’s not science fiction. That’s what AI agents can do for your DevOps workflows.

Over the past 2 years working independently as a DevOps consultant, I’ve seen the same patterns at every client: pipeline failures that need investigation, deployment decisions that require context, and incidents that demand rapid root cause analysis. These aren’t problems that need faster execution. They need reasoning.

That’s when I realized: the CI/CD runner we built is powerful, but it’s missing a brain. So I decided to add one.

Traditional Automation vs. AI Agents

Here’s the fundamental difference:

Your traditional pipeline is like a factory assembly line: efficient and reliable for known workflows, but completely stuck when something unexpected happens.

An AI agent is like a DevOps engineer who can think, investigate, and make decisions based on the full context of your system.

What We’re Building

In this post, I’m going to show you how to build a Pipeline Health Monitor Agent: an AI system that watches your GitHub Actions workflows and autonomously investigates failures.

Here’s what our agent will do:

• Monitor: Watch for GitHub Actions workflow failures via webhooks
• Investigate: Automatically fetch logs, check recent commits, and analyze error patterns
• Reason: Use an LLM (like GPT-4 or Claude) to understand what went wrong
• Report: Post detailed findings to Slack with actionable recommendations
• Learn: Remember similar issues and apply learned patterns

And we’ll do all of this securely. Research shows that 48% of AI-generated code contains vulnerabilities, and I’m going to show you exactly how to validate every action your agent takes.

What You’ll Learn

By the end of this post, you’ll be able to:

• Understand how AI agents differ from traditional automation and when to use each
• Build a working DevOps AI agent using LangChain and LangGraph
• Integrate the agent with your existing GitHub Actions workflows
• Implement security validation layers to prevent AI-generated vulnerabilities

We’ll build this progressively: starting with the core agent, adding GitHub Actions integration, and then hardening it with security layers. Every code example will be complete and runnable.

The core philosophy: AI agents augment your pipeline, they don’t replace it. You’ll still have your traditional CI/CD workflows. The agent just makes them smarter.

Let’s start by understanding what AI agents actually are and how they work.

Understanding AI Agents: The 4 Core Components

Before we start coding, you need to understand what makes an AI agent fundamentally different from a script or a traditional automation workflow.

A traditional pipeline is a sequence of commands. An AI agent is a reasoning loop.

The Agent Loop

Every AI agent operates in a continuous cycle:

┌─────────────────────────────────────────────────────┐
│                                                     │
│  Observe → Reason → Plan → Act → Observe (repeat)   │
│                                                     │
└─────────────────────────────────────────────────────┘

Here’s what happens when your GitHub Actions workflow fails:

1. Observe: Agent receives webhook notification about pipeline failure
2. Reason: LLM analyzes the error message and context
3. Plan: Agent decides which tools to use (check logs, git history, search issues)
4. Act: Agent executes those tools and gathers information
5. Observe: Agent reviews tool outputs and repeats the cycle until it has an answer

This is completely different from your CI/CD runner, which executes steps linearly and stops when something fails.

The 4 Core Components

Every AI agent is built from these four pieces:

1. The LLM (Brain)

The Large Language Model is the decision-making engine. It takes in context (pipeline logs, error messages, git history) and decides what to do next.

Think of it as the “thinking” part. When your pipeline fails with a database connection error, the LLM reasons: “This could be a configuration issue, a networking problem, or resource exhaustion. I should check recent config changes first, then network logs, then resource usage.”

Common choices: GPT-4, Claude 3.5 Sonnet, GPT-3.5 (cheaper for simple tasks)

2. Tools (Hands)

Tools are functions the agent can call to interact with the world. For DevOps, these might be:

• get_github_logs(workflow_id) - Fetch pipeline logs
• analyze_recent_commits(repo, hours) - Check recent code changes
• search_similar_issues(error_message) - Find related GitHub issues
• get_docker_status(container_id) - Check container health
• query_prometheus(metric, timerange) - Get monitoring data

The LLM decides which tools to call and when. You just define what each tool does.

3. Memory

Agents need two types of memory:

Short-term memory (conversation history): The current investigation. “I checked the logs and found a connection error. Then I checked recent commits and found a database config change.”

Long-term memory (learned patterns): Historical knowledge. “The last three times we saw Connection refused on port 5432, it was because the connection pool size was too small.”

For our pipeline monitor, we’ll start with short-term memory. Long-term memory requires a vector database (we’ll save that for a future post).

4. Prompts (Instructions)

The prompt is how you tell the agent what its job is and how to behave. A good DevOps agent prompt includes:

• Role definition: “You are a DevOps AI agent that investigates pipeline failures.”
• Context: “The system runs on Kubernetes in AWS. Database is PostgreSQL. Cache is Redis.”
• Constraints: “Never execute destructive commands. Always explain your reasoning.”
• Output format: “Provide a root cause analysis with suggested fixes.”

Prompt engineering is critical. A vague prompt like “debug the issue” will give you vague results. A specific prompt with context will give you actionable insights.

How It All Works Together

Here’s a concrete example of the agent loop in action:

Pipeline failure detected
    ↓
Agent observes: "Workflow #1234 failed with exit code 1"
    ↓
LLM reasons: "Exit code 1 is generic. I need more information."
    ↓
Agent plans: "Call get_github_logs() to see the actual error"
    ↓
Agent acts: Fetches logs, finds "psycopg2.OperationalError: could not connect to server"
    ↓
LLM reasons: "Database connection failure. Could be config, network, or resources."
    ↓
Agent plans: "Check recent commits for database config changes"
    ↓
Agent acts: Calls analyze_recent_commits(), finds commit changing DATABASE_URL
    ↓
LLM reasons: "Root cause identified. Recent commit broke database connection."
    ↓
Agent outputs: Detailed report with commit hash, explanation, and fix suggestion

When to Use AI Agents vs. Traditional Automation

Not every problem needs an AI agent. Here’s when each makes sense:

Use traditional CI/CD automation when:

• The workflow is fully deterministic
• You know all possible failure modes
• Speed and cost are critical
• Zero tolerance for unexpected behavior

Use AI agents when:

• Failures require investigation and reasoning
• Context matters (recent changes, system state, historical patterns)
• The problem space is too large for explicit if-then rules
• You need adaptive behavior

Examples:

Traditional automation: “If tests fail, don’t deploy” (simple rule)

AI agent: “Tests failed. Analyze which tests, check if they’re flaky, review recent code changes, determine if this is a real issue or infrastructure problem, suggest next steps” (complex reasoning)

What We’re Building Next

Now that you understand the components, we’re going to build a Pipeline Health Monitor Agent that uses:

• LLM: GPT-4 or Claude for reasoning
• Tools: GitHub API, log analysis, issue search
• Memory: Conversation history for multi-step investigation
• Prompts: DevOps-specific instructions with infrastructure context

In the next section, we’ll write the actual code.

Building Version 1: Pipeline Health Monitor Agent

Now we’re going to build a working AI agent that monitors your GitHub Actions workflows and investigates failures. This is production-ready code that you can deploy today.

What Our Agent Will Do

When a GitHub Actions workflow fails, our agent will:

• Receive a webhook notification with the workflow ID
• Fetch the workflow logs from GitHub
• Analyze recent commits to find what changed
• Search existing GitHub issues for similar errors
• Use an LLM (GPT-4, Claude, or others via OpenRouter) to reason about the root cause
• Generate a detailed report with recommendations

Let’s build it step by step.

Installation and Setup

First, install uv if you don’t have it already:

# On macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

# On Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Create a new project directory and set up a virtual environment:

mkdir pipeline-agent
cd pipeline-agent

# Create virtual environment with uv
uv venv

# Activate the virtual environment
# On macOS/Linux:
source .venv/bin/activate

# On Windows:
.venv\Scripts\activate

Install the required dependencies using uv:

uv pip install langchain langchain-openai requests python-dotenv

Set up your environment variables in a .env file.

Option 1: Using OpenAI directly

OPENAI_API_KEY=your_openai_api_key_here
GITHUB_TOKEN=your_github_personal_access_token
GITHUB_REPO=username/repository
USE_OPENROUTER=false

Option 2: Using OpenRouter (recommended for cost savings)

OPENROUTER_API_KEY=your_openrouter_api_key_here
GITHUB_TOKEN=your_github_personal_access_token
GITHUB_REPO=username/repository
USE_OPENROUTER=true
MODEL_NAME=anthropic/claude-3.5-sonnet  # or openai/gpt-4, google/gemini-pro, etc.

Why OpenRouter?

• Access multiple LLM providers through one API
• Often cheaper than going direct (they negotiate bulk rates)
• Easy to switch between models without changing code
• Get API key at: https://openrouter.ai/

Step 1: Define the Tools

Tools are functions the agent can call. Each tool is decorated with @tool and includes a docstring that tells the LLM what it does.

# agent_investigator.py
import os
import requests
from datetime import datetime, timedelta
from typing import Optional
from langchain.tools import tool
from dotenv import load_dotenv

load_dotenv()

GITHUB_TOKEN = os.getenv("GITHUB_TOKEN")
GITHUB_REPO = os.getenv("GITHUB_REPO")
HEADERS = {
    "Authorization": f"token {GITHUB_TOKEN}",
    "Accept": "application/vnd.github.v3+json"
}

@tool
def get_workflow_logs(workflow_run_id: str) -> str:
    """
    Fetch logs from a failed GitHub Actions workflow run.

    Args:
        workflow_run_id: The GitHub Actions workflow run ID

    Returns:
        String containing the workflow logs
    """
    try:
        # Get workflow run details
        run_url = f"https://api.github.com/repos/{GITHUB_REPO}/actions/runs/{workflow_run_id}"
        run_response = requests.get(run_url, headers=HEADERS)
        run_response.raise_for_status()
        run_data = run_response.json()

        # Get jobs for this workflow run
        jobs_url = f"{run_url}/jobs"
        jobs_response = requests.get(jobs_url, headers=HEADERS)
        jobs_response.raise_for_status()
        jobs_data = jobs_response.json()

        # Extract logs from failed jobs
        logs = []
        logs.append(f"Workflow: {run_data['name']}")
        logs.append(f"Status: {run_data['conclusion']}")
        logs.append(f"Started: {run_data['created_at']}")
        logs.append(f"Branch: {run_data['head_branch']}\n")

        for job in jobs_data['jobs']:
            if job['conclusion'] == 'failure':
                logs.append(f"\nFailed Job: {job['name']}")
                logs.append(f"Conclusion: {job['conclusion']}")

                # Get job logs
                log_url = f"https://api.github.com/repos/{GITHUB_REPO}/actions/jobs/{job['id']}/logs"
                log_response = requests.get(log_url, headers=HEADERS)

                if log_response.status_code == 200:
                    # Extract last 50 lines (most relevant errors are at the end)
                    log_lines = log_response.text.split('\n')
                    relevant_logs = log_lines[-50:]
                    logs.append("\nLast 50 lines of logs:")
                    logs.append('\n'.join(relevant_logs))

        return '\n'.join(logs)

    except requests.exceptions.RequestException as e:
        return f"Error fetching workflow logs: {str(e)}"


@tool
def analyze_recent_commits(hours: int = 24) -> str:
    """
    Analyze recent commits to the repository that might have caused the failure.

    Args:
        hours: Number of hours to look back (default: 24)

    Returns:
        String containing recent commits with author, message, and files changed
    """
    try:
        since = (datetime.utcnow() - timedelta(hours=hours)).isoformat() + 'Z'
        commits_url = f"https://api.github.com/repos/{GITHUB_REPO}/commits"
        params = {'since': since, 'per_page': 10}

        response = requests.get(commits_url, headers=HEADERS, params=params)
        response.raise_for_status()
        commits = response.json()

        if not commits:
            return f"No commits found in the last {hours} hours."

        result = [f"Recent commits (last {hours} hours):\n"]

        for commit in commits:
            sha = commit['sha'][:7]
            author = commit['commit']['author']['name']
            message = commit['commit']['message'].split('\n')[0]  # First line only
            date = commit['commit']['author']['date']

            # Get files changed in this commit
            commit_detail_url = f"https://api.github.com/repos/{GITHUB_REPO}/commits/{commit['sha']}"
            commit_response = requests.get(commit_detail_url, headers=HEADERS)
            commit_data = commit_response.json()

            files_changed = [f['filename'] for f in commit_data.get('files', [])]

            result.append(f"\nCommit {sha} by {author} ({date})")
            result.append(f"Message: {message}")
            result.append(f"Files changed: {', '.join(files_changed[:5])}")  # First 5 files
            if len(files_changed) > 5:
                result.append(f"... and {len(files_changed) - 5} more files")

        return '\n'.join(result)

    except requests.exceptions.RequestException as e:
        return f"Error analyzing commits: {str(e)}"


@tool
def search_similar_issues(error_keywords: str) -> str:
    """
    Search GitHub issues for similar error messages or problems.

    Args:
        error_keywords: Keywords from the error message to search for

    Returns:
        String containing relevant GitHub issues and their solutions
    """
    try:
        # Search issues in the repository
        search_url = "https://api.github.com/search/issues"
        query = f"repo:{GITHUB_REPO} {error_keywords} is:issue"
        params = {'q': query, 'sort': 'relevance', 'per_page': 5}

        response = requests.get(search_url, headers=HEADERS, params=params)
        response.raise_for_status()
        issues = response.json()

        if issues['total_count'] == 0:
            return f"No similar issues found for keywords: {error_keywords}"

        result = [f"Found {issues['total_count']} similar issues:\n"]

        for issue in issues['items'][:5]:
            result.append(f"\n#{issue['number']}: {issue['title']}")
            result.append(f"State: {issue['state']}")
            result.append(f"URL: {issue['html_url']}")

            # Get first comment if issue is closed (might contain solution)
            if issue['state'] == 'closed' and issue['comments'] > 0:
                comments_url = issue['comments_url']
                comments_response = requests.get(comments_url, headers=HEADERS)
                comments = comments_response.json()
                if comments:
                    first_comment = comments[0]['body'][:200]  # First 200 chars
                    result.append(f"Solution hint: {first_comment}...")

        return '\n'.join(result)

    except requests.exceptions.RequestException as e:
        return f"Error searching issues: {str(e)}"

Step 2: Create the Agent with LLM Provider Support

Now we’ll create the agent with support for both OpenAI and OpenRouter:

from langchain.agents import create_openai_tools_agent, AgentExecutor
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder

def get_llm():
    """
    Initialize the LLM based on environment configuration.
    Supports both OpenAI directly and OpenRouter.
    """
    use_openrouter = os.getenv("USE_OPENROUTER", "false").lower() == "true"

    if use_openrouter:
        # Using OpenRouter for access to multiple models
        api_key = os.getenv("OPENROUTER_API_KEY")
        model_name = os.getenv("MODEL_NAME", "anthropic/claude-3.5-sonnet")

        llm = ChatOpenAI(
            model=model_name,
            openai_api_key=api_key,
            openai_api_base="https://openrouter.ai/api/v1",
            temperature=0,
            default_headers={
                "HTTP-Referer": "https://github.com/your-username/pipeline-agent",
                "X-Title": "Pipeline Health Monitor Agent"
            }
        )
        print(f"Using OpenRouter with model: {model_name}")
    else:
        # Using OpenAI directly
        api_key = os.getenv("OPENAI_API_KEY")
        llm = ChatOpenAI(
            model="gpt-4",
            temperature=0,
            openai_api_key=api_key
        )
        print("Using OpenAI GPT-4")

    return llm

# Initialize the LLM
llm = get_llm()

# Define the system prompt
system_prompt = """You are an expert DevOps AI agent that investigates CI/CD pipeline failures.

Your role is to:
1. Analyze workflow logs to identify the root cause of failures
2. Examine recent code changes that might have introduced issues
3. Search for similar problems in the issue tracker
4. Provide a clear, actionable root cause analysis

When analyzing failures:
- Focus on the actual error messages, not just symptoms
- Consider recent code changes as potential causes
- Look for patterns in similar past issues
- Be specific about what broke and why
- Suggest concrete fixes, not vague advice

Your investigation should be thorough but concise. Developers need actionable insights, not lengthy explanations.

Output format:
**Root Cause**: [One sentence summary]
**Evidence**: [Key findings from logs/commits/issues]
**Recommendation**: [Specific steps to fix]
**Related Issues**: [Links to similar problems if found]
"""

# Create the prompt template
prompt = ChatPromptTemplate.from_messages([
    ("system", system_prompt),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad"),
])

# Create the agent
tools = [get_workflow_logs, analyze_recent_commits, search_similar_issues]
agent = create_openai_tools_agent(llm, tools, prompt)

# Create the agent executor
agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    max_iterations=5,
    handle_parsing_errors=True
)

Step 3: Run the Investigation

Finally, we create a function to trigger the investigation:

def investigate_failure(workflow_run_id: str) -> dict:
    """
    Investigate a failed GitHub Actions workflow.

    Args:
        workflow_run_id: The GitHub Actions workflow run ID

    Returns:
        Dict containing the investigation result
    """
    print(f"\nStarting investigation for workflow run {workflow_run_id}...")
    print("=" * 60)

    input_text = f"""A GitHub Actions workflow has failed (run ID: {workflow_run_id}).

Please investigate this failure by:
1. Fetching and analyzing the workflow logs
2. Checking recent commits for changes that might have caused this
3. Searching for similar issues that might provide insights

Provide a comprehensive root cause analysis with specific recommendations."""

    try:
        result = agent_executor.invoke({"input": input_text})

        print("\n" + "=" * 60)
        print("INVESTIGATION COMPLETE")
        print("=" * 60)
        print(result['output'])

        return {
            "success": True,
            "workflow_run_id": workflow_run_id,
            "analysis": result['output']
        }

    except Exception as e:
        print(f"\nError during investigation: {str(e)}")
        return {
            "success": False,
            "workflow_run_id": workflow_run_id,
            "error": str(e)
        }


if __name__ == "__main__":
    import sys

    if len(sys.argv) < 2:
        print("Usage: python agent_investigator.py <workflow_run_id>")
        sys.exit(1)

    workflow_run_id = sys.argv[1]
    investigate_failure(workflow_run_id)

Model Recommendations via OpenRouter

Here are some good model choices for DevOps investigations:

For best reasoning (higher cost):

• anthropic/claude-3.5-sonnet - Excellent at technical analysis
• openai/gpt-4-turbo - Strong general reasoning
• google/gemini-pro-1.5 - Good for long context (helpful with large logs)

For cost efficiency (lower cost):

• anthropic/claude-3-haiku - Fast and cheap, good for simple failures
• openai/gpt-3.5-turbo - Decent reasoning, very affordable
• meta-llama/llama-3.1-70b-instruct - Open source, cost-effective

Cost comparison per investigation:

• GPT-4: ~$0.15-0.30
• Claude 3.5 Sonnet: ~$0.10-0.20
• GPT-3.5: ~$0.02-0.05
• Llama 3.1 70B: ~$0.01-0.03

How It Works

Let’s walk through what happens when you run this:

1. You trigger the agent: python agent_investigator.py 12345678

2. Agent receives the task: “Investigate workflow run 12345678”

3. LLM decides first action: “I should fetch the workflow logs to see what failed”

4. Agent calls get_workflow_logs(): Returns the last 50 lines of failed job logs

5. LLM analyzes logs: “I see a database connection error. Let me check recent commits for database config changes”

6. Agent calls analyze_recent_commits(): Returns commits from the last 24 hours

7. LLM finds suspicious commit: “Commit abc123 changed database.yml. Let me search for similar issues”

8. Agent calls search_similar_issues(): Finds issue #42 about database connection problems

9. LLM synthesizes findings: Produces a final report with root cause and fix

The entire process takes 10-30 seconds depending on the complexity.

Example Output

Here’s what the agent produces for a real failure:

Root Cause: Database connection pool exhaustion caused by recent increase in concurrent workers without adjusting max_connections setting.

Evidence:
- Workflow logs show "psycopg2.OperationalError: FATAL: sorry, too many clients already"
- Commit d4e5f6a (2 hours ago) changed worker count from 4 to 16 in deploy.yml
- Issue #127 documented same error when worker count was increased last month

Recommendation:
1. Increase PostgreSQL max_connections from 100 to 200 in database config
2. Or reduce worker count back to 8 as a temporary fix
3. Add connection pooling with PgBouncer for better resource management

Related Issues:
- #127: Database connection errors after scaling workers
- #89: PostgreSQL connection pool configuration guide

This is exactly what you need: the root cause, evidence, and actionable fixes.

Key Design Decisions

Why max_iterations=5? Prevents infinite loops. Most investigations complete in 3-4 iterations.

Why last 50 lines of logs? Error messages are typically at the end. Sending full logs wastes tokens and costs money.

Why temperature=0? We want deterministic, factual analysis. Higher temperature adds creativity, which we don’t need for debugging.

Why support OpenRouter? Gives you flexibility to switch models based on cost and performance. Claude 3.5 Sonnet often performs better than GPT-4 for technical debugging at a lower price.

In the next section, we’ll integrate this agent with GitHub Actions so it runs automatically when workflows fail.

GitHub Actions Integration

Now that we have a working agent, let’s integrate it with GitHub Actions so it automatically investigates failures. We’ll use GitHub’s workflow events to trigger our agent whenever a pipeline fails.

Architecture Overview

Here’s how the integration works:

GitHub Actions Workflow Fails
    ↓
GitHub triggers workflow_run event
    ↓
Our "Investigate Failure" workflow runs
    ↓
Calls agent_investigator.py with workflow ID
    ↓
Agent investigates and generates report
    ↓
Posts results to GitHub issue or Slack

Step 1: Set Up GitHub Secrets

First, add your API keys to GitHub repository secrets:

1. Go to your repository on GitHub
2. Click Settings > Secrets and variables > Actions
3. Click New repository secret
4. Add these secrets:

OPENAI_API_KEY (or OPENROUTER_API_KEY)
GITHUB_TOKEN (automatically provided by GitHub Actions)
SLACK_WEBHOOK_URL (optional, for notifications)

For OpenRouter users, also add:

USE_OPENROUTER=true
MODEL_NAME=anthropic/claude-3.5-sonnet

Step 2: Create the Investigation Workflow

Create a new file .github/workflows/investigate-failures.yml:

name: AI Agent - Investigate Failures

on:
  workflow_run:
    workflows: ["*"]  # Monitor all workflows
    types:
      - completed

jobs:
  investigate:
    # Only run if the workflow failed
    if: $
    runs-on: ubuntu-latest

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install uv
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          echo "$HOME/.cargo/bin" >> $GITHUB_PATH

      - name: Create virtual environment and install dependencies
        run: |
          uv venv
          source .venv/bin/activate
          uv pip install langchain langchain-openai requests python-dotenv

      - name: Run AI investigation
        env:
          GITHUB_TOKEN: $
          GITHUB_REPO: $
          OPENAI_API_KEY: $
          OPENROUTER_API_KEY: $
          USE_OPENROUTER: $
          MODEL_NAME: $
        run: |
          source .venv/bin/activate
          python agent_investigator.py $

      - name: Post results to GitHub issue
        if: always()
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');

            // Read the investigation results
            const workflowName = '$';
            const workflowUrl = '$';
            const runId = '$';

            // Create or update issue with findings
            const title = `Pipeline Failure: ${workflowName}`;
            const body = `## Automated Investigation Report

**Workflow**: [${workflowName}](${workflowUrl})
**Run ID**: ${runId}
**Branch**: $
**Commit**: $

### Investigation Results

The AI agent has completed its investigation. Check the workflow logs for detailed analysis.

**Next Steps**:
1. Review the root cause analysis above
2. Check the recommended fixes
3. Review related issues if any were found
4. Apply the fix and re-run the workflow

---
*This issue was automatically created by the Pipeline Health Monitor AI Agent*
`;

            // Search for existing open issue
            const issues = await github.rest.issues.listForRepo({
              owner: context.repo.owner,
              repo: context.repo.repo,
              state: 'open',
              labels: ['pipeline-failure', 'ai-investigated']
            });

            const existingIssue = issues.data.find(issue =>
              issue.title.includes(workflowName)
            );

            if (existingIssue) {
              // Update existing issue
              await github.rest.issues.createComment({
                owner: context.repo.owner,
                repo: context.repo.repo,
                issue_number: existingIssue.number,
                body: `## New Failure Detected\n\n${body}`
              });
            } else {
              // Create new issue
              await github.rest.issues.create({
                owner: context.repo.owner,
                repo: context.repo.repo,
                title: title,
                body: body,
                labels: ['pipeline-failure', 'ai-investigated']
              });
            }

How It Works in Production

Once deployed, here’s what happens automatically:

1. Developer pushes code that breaks a test
2. CI pipeline fails (tests, build, deployment, etc.)
3. GitHub triggers the workflow_run event
4. Investigation workflow starts within seconds
5. Agent fetches logs, analyzes commits, searches issues
6. LLM reasons about the root cause
7. Results posted to GitHub issue and Slack
8. Developer sees detailed analysis with fix recommendations

All of this happens in 30-60 seconds after the failure.

Cost Considerations

Each investigation costs approximately:

• GPT-4: $0.15-0.30 per investigation
• Claude 3.5 Sonnet (via OpenRouter): $0.10-0.20
• GPT-3.5: $0.02-0.05

For a team with:

• 20 pipeline failures per day
• Using Claude 3.5 Sonnet ($0.15 average)

Monthly cost: 20 × $0.15 × 30 = $90

Compare this to:

• Developer time investigating failures: 30 min × 20 failures = 10 hours/day
• At $100/hour = $1,000/day saved

The ROI is clear.

Security Validation: The 48% Vulnerability Problem

Here’s the uncomfortable truth: research shows that 48% of AI-generated code contains vulnerabilities. In some studies, 60% of AI suggestions for financial services contained high-severity security flaws.

As DevOps consultants, we can’t afford to blindly trust AI-generated recommendations. Our agent has read access to logs, commits, and issues, but what if we extend it to execute fixes automatically? We need layers of security validation.

The Real Security Risks

Before we dive into solutions, let’s understand what can go wrong:

Prompt Injection Attacks: Google’s security team demonstrated a real exploit where hidden HTML comments in a dependency’s README convinced a build agent that a malicious package was legitimate. The agent shipped the malicious code to production.

Hallucinated Commands: An LLM might confidently suggest running kubectl delete deployment production when it meant to suggest kubectl delete pod production-5f6h8.

Information Leakage: Agents with access to logs might inadvertently expose secrets, API keys, or sensitive data when posting to public channels.

Shadow AI: Developers creating custom agents without proper governance, leading to unauthorized automation running in your pipelines.

Let’s build defenses against all of these.

Layer 1: Restrict Agent Permissions

The principle of least privilege applies to AI agents just like any other system component.

Our current agent only has read-only access:

# Current tools - all read-only
tools = [
    get_workflow_logs,       # Read GitHub logs
    analyze_recent_commits,  # Read git history
    search_similar_issues    # Read GitHub issues
]

This is intentional. Investigation does not require execution.

Layer 2: Secrets Detection

Never let the agent expose secrets in logs or notifications.

Create a secrets scanner:

# secrets_scanner.py
import re
from typing import List, Tuple

class SecretsScanner:
    """Detect and redact secrets from agent outputs."""

    PATTERNS = {
        'aws_key': r'AKIA[0-9A-Z]{16}',
        'github_token': r'gh[pousr]_[A-Za-z0-9_]{36,255}',
        'generic_api_key': r'api[_-]?key["\']?\s*[:=]\s*["\']?([a-zA-Z0-9_\-]{20,})',
        'password': r'password["\']?\s*[:=]\s*["\']?([^\s"\']{8,})',
        'private_key': r'-----BEGIN (RSA |OPENSSH )?PRIVATE KEY-----',
        'jwt': r'eyJ[A-Za-z0-9-_=]+\.eyJ[A-Za-z0-9-_=]+\.?[A-Za-z0-9-_.+/=]*',
        'connection_string': r'(postgres|mysql|mongodb)://[^:]+:[^@]+@',
    }

    @staticmethod
    def scan(text: str) -> Tuple[bool, List[str]]:
        """
        Scan text for secrets.

        Args:
            text: Text to scan

        Returns:
            Tuple of (has_secrets, list of secret types found)
        """
        found_secrets = []

        for secret_type, pattern in SecretsScanner.PATTERNS.items():
            if re.search(pattern, text, re.IGNORECASE):
                found_secrets.append(secret_type)

        return (len(found_secrets) > 0, found_secrets)

    @staticmethod
    def redact(text: str) -> str:
        """
        Redact secrets from text.

        Args:
            text: Text to redact

        Returns:
            Text with secrets replaced by [REDACTED]
        """
        redacted = text

        for secret_type, pattern in SecretsScanner.PATTERNS.items():
            redacted = re.sub(pattern, f'[REDACTED:{secret_type.upper()}]', redacted, flags=re.IGNORECASE)

        return redacted


# Usage in agent output
def safe_output(text: str) -> str:
    """Process agent output to remove secrets before displaying."""
    scanner = SecretsScanner()
    has_secrets, secret_types = scanner.scan(text)

    if has_secrets:
        print(f"WARNING: Detected secrets in output: {', '.join(secret_types)}")
        return scanner.redact(text)

    return text

Update the investigation function to use secrets scanning:

def investigate_failure(workflow_run_id: str) -> dict:
    """Investigate a failed GitHub Actions workflow with secret protection."""
    # ... existing code ...

    try:
        result = agent_executor.invoke({"input": input_text})

        # Scan for secrets before outputting
        safe_analysis = safe_output(result['output'])

        print("\n" + "=" * 60)
        print("INVESTIGATION COMPLETE")
        print("=" * 60)
        print(safe_analysis)

        return {
            "success": True,
            "workflow_run_id": workflow_run_id,
            "analysis": safe_analysis
        }
    except Exception as e:
        return {
            "success": False,
            "workflow_run_id": workflow_run_id,
            "error": str(e)
        }

Layer 3: Audit Trail

Log every agent decision for security review and debugging.

# audit_logger.py
import json
from datetime import datetime
from pathlib import Path
from typing import Dict, Any

class AuditLogger:
    """Log all agent actions for security auditing."""

    def __init__(self, log_dir: str = ".agent_logs"):
        self.log_dir = Path(log_dir)
        self.log_dir.mkdir(exist_ok=True)

    def log_investigation(self, event_data: Dict[str, Any]):
        """
        Log an investigation event.

        Args:
            event_data: Dictionary containing event details
        """
        timestamp = datetime.utcnow().isoformat()
        log_entry = {
            "timestamp": timestamp,
            "event_type": "investigation",
            **event_data
        }

        # Log to daily file
        log_file = self.log_dir / f"audit_{datetime.utcnow().strftime('%Y-%m-%d')}.jsonl"

        with open(log_file, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

    def log_tool_call(self, tool_name: str, args: Dict, result: Any, duration: float):
        """Log a tool call."""
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "event_type": "tool_call",
            "tool": tool_name,
            "arguments": args,
            "result_preview": str(result)[:200],
            "duration_seconds": duration
        }

        log_file = self.log_dir / f"audit_{datetime.utcnow().strftime('%Y-%m-%d')}.jsonl"

        with open(log_file, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

    def log_security_event(self, event_type: str, details: Dict[str, Any]):
        """Log a security-related event."""
        log_entry = {
            "timestamp": datetime.utcnow().isoformat(),
            "event_type": "security",
            "security_event": event_type,
            **details
        }

        log_file = self.log_dir / f"security_{datetime.utcnow().strftime('%Y-%m-%d')}.jsonl"

        with open(log_file, 'a') as f:
            f.write(json.dumps(log_entry) + '\n')

Layer 4: Rate Limiting and Cost Controls

Prevent runaway costs and API abuse:

# rate_limiter.py
import time
from collections import deque
from datetime import datetime, timedelta

class RateLimiter:
    """Rate limit agent executions to prevent abuse and control costs."""

    def __init__(self, max_investigations_per_hour: int = 20):
        self.max_per_hour = max_investigations_per_hour
        self.investigation_times = deque()

    def can_investigate(self) -> bool:
        """Check if we can run another investigation."""
        now = datetime.utcnow()
        cutoff = now - timedelta(hours=1)

        # Remove investigations older than 1 hour
        while self.investigation_times and self.investigation_times[0] < cutoff:
            self.investigation_times.popleft()

        return len(self.investigation_times) < self.max_per_hour

    def record_investigation(self):
        """Record that an investigation occurred."""
        self.investigation_times.append(datetime.utcnow())

    def time_until_next_allowed(self) -> int:
        """Get seconds until next investigation is allowed."""
        if self.can_investigate():
            return 0

        oldest = self.investigation_times[0]
        time_until_allowed = (oldest + timedelta(hours=1)) - datetime.utcnow()
        return int(time_until_allowed.total_seconds())

Security Checklist

Before deploying your AI agent to production, verify:

• Agent has minimum required permissions (read-only by default)
• All commands validated before execution
• Secrets scanner active on all outputs
• Audit logging enabled and monitored
• Rate limiting configured
• GitHub tokens scoped correctly (no admin access)
• LLM API keys stored in secrets, not code
• No secrets committed to repository
• Slack webhooks use incoming webhook URLs only
• Agent cannot modify production without approval

Real-World Security Scenario

Here’s how these layers work together:

1. Agent investigates failure and LLM suggests: kubectl delete pod production-db-0
2. Command validator catches this: “APPROVAL REQUIRED: Command requires human approval”
3. Agent posts recommendation to GitHub issue instead of executing
4. Secrets scanner detects database connection string in logs and redacts it
5. Audit logger records the attempted command and approval requirement
6. Human reviews the recommendation and decides whether to execute
7. If approved, human runs command manually with full context

The agent accelerates investigation but humans retain control over critical actions.

Practical Tips and Common Pitfalls

After building and running AI agents for DevOps investigations, I’ve learned what works and what doesn’t. Here are the hard-earned lessons that will save you time and money.

Prompt Engineering Best Practices

Your prompt is the most important part of your agent. A vague prompt gives vague results. A specific prompt with context gives actionable insights.

Bad Prompt:

system_prompt = """You are an AI agent. Debug the issue."""

Why it fails: Too generic, no context, no output format.

Good Prompt:

system_prompt = """You are an expert DevOps AI agent that investigates CI/CD pipeline failures.

Infrastructure context:
- Python microservices running on Kubernetes in AWS EKS
- PostgreSQL 14 database with connection pooling
- Redis for caching
- GitHub Actions for CI/CD

Your role is to:
1. Analyze workflow logs to identify the root cause of failures
2. Examine recent code changes that might have introduced issues
3. Search for similar problems in the issue tracker
4. Provide a clear, actionable root cause analysis

When analyzing failures:
- Focus on the actual error messages, not just symptoms
- Consider recent code changes as potential causes
- Look for patterns in similar past issues
- Be specific about what broke and why
- Suggest concrete fixes, not vague advice

Output format:
**Root Cause**: [One sentence summary]
**Evidence**: [Key findings from logs/commits/issues]
**Recommendation**: [Specific steps to fix]
**Related Issues**: [Links to similar problems if found]
"""

Why it works: Infrastructure context, clear role, specific instructions, defined output format.

Common Pitfalls and Solutions

Pitfall 1: Agent Loops Infinitely

Symptom: Agent keeps calling tools without making progress.

Solution: Set max_iterations:

agent_executor = AgentExecutor(
    agent=agent,
    tools=tools,
    verbose=True,
    max_iterations=5,  # Stop after 5 iterations
    handle_parsing_errors=True
)

Pitfall 2: Costs Spiral Out of Control

Symptom: Your OpenAI bill is $500 for 100 investigations.

Cause: Using GPT-4 for everything, not optimizing token usage.

Solution: Use the right model for the task:

def get_llm(task_complexity: str = "medium"):
    """Choose LLM based on task complexity."""

    if task_complexity == "simple":
        # Use cheaper model for simple log analysis
        model = "gpt-3.5-turbo"  # $0.002 per investigation
    elif task_complexity == "medium":
        model = "anthropic/claude-3.5-sonnet"  # $0.15 per investigation
    else:  # complex
        model = "openai/gpt-4"  # $0.30 per investigation

    return ChatOpenAI(model=model, temperature=0)

Cost comparison:

• GPT-4: $0.30 per investigation
• Claude 3.5 Sonnet: $0.15 per investigation
• GPT-3.5: $0.02 per investigation

For 100 investigations/month:

• All GPT-4: $30
• All GPT-3.5: $2
• Mixed (80% GPT-3.5, 20% GPT-4): $6.80

Pitfall 3: Secrets Leak in Logs

Symptom: API keys visible in agent output.

Solution: Always scan output (from the security section):

from secrets_scanner import safe_output

result = agent_executor.invoke({"input": input_text})
safe_result = safe_output(result['output'])  # Redacts secrets

Performance Benchmarks

From my production deployments:

Investigation time:

• Simple failures (import errors): 10-15 seconds
• Medium complexity (config issues): 20-30 seconds
• Complex failures (race conditions): 45-60 seconds

Accuracy:

• Correct root cause identified: 78% of cases
• Helpful suggestions even when wrong: 92% of cases
• Completely useless output: 8% of cases

Cost per investigation:

• GPT-3.5: $0.02-0.05
• Claude 3.5 Sonnet: $0.10-0.20
• GPT-4: $0.15-0.30

Developer time saved:

• Average investigation time (manual): 25 minutes
• Average investigation time (agent): 30 seconds
• Time saved: 24.5 minutes per failure

For 20 failures/day: 490 minutes = 8+ hours saved daily.

Quick Reference: Dos and Don’ts

DO:

• Set max_iterations to prevent loops
• Add timeouts to all API calls
• Scan outputs for secrets
• Log all agent decisions
• Use structured output formats
• Cache frequent queries
• Choose models based on complexity
• Test prompts in isolation first

DON’T:

• Give agents write access without validation
• Trust AI-generated commands blindly
• Send full logs (use last 50 lines)
• Use GPT-4 for everything (cost optimization)
• Ignore rate limits
• Commit API keys to git
• Skip error handling
• Deploy without testing

Next Steps and Extensions

You’ve built a working AI agent that automatically investigates pipeline failures. But this is just the beginning. Here are practical ways to extend and improve your agent.

What You’ve Built

Let’s recap what your agent can do:

• Monitor GitHub Actions workflows automatically
• Investigate failures within 30 seconds
• Fetch and analyze workflow logs
• Examine recent code changes
• Search for similar issues
• Generate root cause analysis with recommendations
• Redact secrets from outputs
• Log all actions for audit
• Rate limit to control costs
• Post results to GitHub issues

Extension Ideas

1. Multi-Agent System

Create specialist agents for different tasks:

# Build Agent: Optimizes build performance
build_agent = create_agent(
    tools=[analyze_build_logs, suggest_caching, optimize_dependencies],
    role="Build Optimization Specialist"
)

# Security Agent: Scans for vulnerabilities
security_agent = create_agent(
    tools=[scan_dependencies, check_secrets, validate_configs],
    role="Security Analyst"
)

# Deploy Agent: Manages deployments
deploy_agent = create_agent(
    tools=[check_health, deploy_staging, rollback_if_needed],
    role="Deployment Specialist"
)

2. Kubernetes Integration

Add tools for Kubernetes operations:

@tool
def get_pod_status(namespace: str, pod_name: str) -> str:
    """Get Kubernetes pod status and recent events."""
    pass

@tool
def analyze_pod_logs(namespace: str, pod_name: str) -> str:
    """Fetch and analyze pod logs."""
    pass

3. Learning from History

Implement long-term memory with a vector database:

from langchain.vectorstores import Chroma
from langchain.embeddings import OpenAIEmbeddings

# Store past investigations
vectorstore = Chroma(
    collection_name="investigation_history",
    embedding_function=OpenAIEmbeddings()
)

# When investigating a new failure
similar_cases = vectorstore.similarity_search(
    error_message,
    k=3  # Find 3 most similar past failures
)

This lets your agent learn from experience.

Resources and Further Learning

LangChain Documentation

• LangChain Official Docs
• LangChain Agents Guide
• LangChain Tools Documentation

OpenRouter

• Get API key
• Pricing
• Model comparison

Security Resources

• OWASP LLM Top 10

Final Thoughts

AI agents aren’t replacing DevOps engineers. They’re accelerating investigation, reducing toil, and freeing you to focus on higher-value work.

The agent we built is read-only by design. It investigates and recommends, but humans make the final decisions. This is the right balance for production systems in 2025.

Start small:

1. Deploy the read-only investigation agent
2. Monitor its accuracy for a few weeks
3. Tune prompts based on results
4. Gradually add more capabilities
5. Always maintain human oversight

Over the past 2 years as a DevOps consultant, I’ve seen teams waste countless hours on repetitive failure investigations. This agent solves that problem.

The code is production-ready. The security is enterprise-grade. The cost is negligible compared to developer time saved.

What are you waiting for? Give your CI/CD pipeline a brain.

Want to Learn More?

If you’re interested in deepening your DevOps and systems programming knowledge, check out Educative.io’s Unlimited Plan - it’s an excellent resource for hands-on learning with interactive courses.

If you found this helpful, share it on X and tag me @muhammad_o7 - I’d love to hear your thoughts! You can also connect with me on LinkedIn.

Need Help? I’m available for Python and DevOps consulting. If you need help with CI/CD, automation, infrastructure, or AI agents for your DevOps workflows, reach out via email or DM me on X/Twitter.

That changed when I needed to set up CI/CD for an air-gapped environment at work - no access to GitHub Actions or GitLab’s hosted runners. I needed to understand how these tools actually work under the hood so I could build something custom. That’s when I realized: pipeline runners are just orchestration tools that execute jobs in isolated environments.

In this post, I’ll show you how to build a complete CI/CD pipeline runner from scratch in Python. We’ll implement the core features you use every day: stages, parallel execution, job dependencies, and artifact passing. By the end, you’ll understand exactly how GitLab Runner and GitHub Actions work internally.

What Actually IS a CI/CD Pipeline?

Before we start coding, let’s understand what a CI/CD pipeline actually does.

A CI/CD pipeline is an automated workflow that takes your code from commit to deployment. It’s defined as a series of jobs organized into stages:

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - python -m build

test-job:
  stage: test
  script:
    - pytest tests/

deploy-job:
  stage: deploy
  script:
    - ./deploy.sh

When you push code, a pipeline runner (like GitLab Runner or GitHub Actions) does this:

1. Parses the pipeline configuration file
2. Creates a dependency graph of jobs
3. Executes jobs in isolated environments (usually containers)
4. Streams logs back to you in real-time
5. Passes artifacts between jobs
6. Reports success or failure

The key insight: a pipeline runner is just a job orchestrator. It figures out what to run, in what order, and handles the execution.

Let’s build one ourselves.

Understanding Pipeline Components

Every pipeline has three main components:

1. Stages

Stages define the execution order. Jobs in the same stage can run in parallel, but stages run sequentially:

Stage 1: build     → Stage 2: test        → Stage 3: deploy
  [build-app]        [unit-test]             [deploy-prod]
                     [integration-test]

2. Jobs

Jobs are the actual work units. Each job:

• Runs in an isolated environment (container)
• Executes a series of shell commands
• Can depend on other jobs
• Can produce artifacts for other jobs

3. Artifacts

Artifacts are files produced by one job that other jobs need. For example:

• Build job produces dist/ folder
• Test jobs need dist/ to run tests
• Deploy job needs dist/ to deploy

Now that we understand the concepts, let’s start building.

Building Our Pipeline Runner

Version 1: Single Job Executor

Let’s start with the absolute basics - executing a single job in a Docker container:

#!/usr/bin/env python3
"""
Pipeline Runner v1: Single job executor
Executes one job in a Docker container and streams logs
"""

import yaml
import subprocess
import sys
from pathlib import Path


class Job:
    """Represents a single pipeline job."""

    def __init__(self, name, config):
        self.name = name
        self.image = config.get('image', 'python:3.11')
        self.script = config.get('script', [])
        self.stage = config.get('stage', 'test')

    def __repr__(self):
        return f"Job({self.name}, stage={self.stage})"


class JobExecutor:
    """Executes a job in a Docker container."""

    def __init__(self, workspace):
        self.workspace = Path(workspace).resolve()

    def run(self, job):
        """Execute a job and stream output."""
        print(f"\n{'='*60}")
        print(f"[{job.name}] Starting job...")
        print(f"[{job.name}] Image: {job.image}")
        print(f"[{job.name}] Stage: {job.stage}")
        print(f"{'='*60}\n")

        # Combine all script commands into one shell command
        script = ' && '.join(job.script)

        # Build docker run command
        cmd = [
            'docker', 'run',
            '--rm',  # Remove container after execution
            '-v', f'{self.workspace}:/workspace',  # Mount workspace
            '-w', '/workspace',  # Set working directory
            job.image,
            'sh', '-c', script
        ]

        try:
            # Run and stream output in real-time
            process = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
                text=True,
                bufsize=1
            )

            # Stream output line by line
            for line in process.stdout:
                print(f"[{job.name}] {line}", end='')

            process.wait()

            if process.returncode == 0:
                print(f"\n[{job.name}] ✓ Job completed successfully\n")
                return True
            else:
                print(f"\n[{job.name}] ✗ Job failed with exit code {process.returncode}\n")
                return False

        except Exception as e:
            print(f"\n[{job.name}] ✗ Error: {e}\n")
            return False


class Pipeline:
    """Represents a pipeline with jobs."""

    def __init__(self, config_file):
        self.config_file = Path(config_file)
        self.config = self._load_config()
        self.jobs = self._parse_jobs()

    def _load_config(self):
        """Load and parse YAML configuration."""
        with open(self.config_file) as f:
            return yaml.safe_load(f)

    def _parse_jobs(self):
        """Parse jobs from configuration."""
        jobs = []
        for job_name, job_config in self.config.items():
            if job_name != 'stages' and isinstance(job_config, dict):
                jobs.append(Job(job_name, job_config))
        return jobs

    def run(self, workspace='.'):
        """Execute all jobs sequentially."""
        print(f"\nStarting pipeline from {self.config_file}")
        print(f"Found {len(self.jobs)} job(s)\n")

        executor = JobExecutor(workspace)

        for job in self.jobs:
            success = executor.run(job)
            if not success:
                print("Pipeline failed!")
                return False

        print("✓ Pipeline completed successfully!")
        return True


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python runner_v1.py <pipeline.yml>")
        sys.exit(1)

    pipeline = Pipeline(sys.argv[1])
    success = pipeline.run()
    sys.exit(0 if success else 1)

Testing Version 1

Create a simple pipeline config:

# pipeline.yml
build-job:
  image: python:3.11
  script:
    - echo "Building application..."
    - python --version
    - pip install --quiet build
    - echo "Build complete!"

Run it:

python runner_v1.py pipeline.yml

You should see output like:

Starting pipeline from pipeline.yml
Found 1 job(s)

============================================================
[build-job] Starting job...
[build-job] Image: python:3.11
[build-job] Stage: test
============================================================

[build-job] Building application...
[build-job] Python 3.11.9
[build-job] Build complete!

[build-job] ✓ Job completed successfully

✓ Pipeline completed successfully!

Great! We can execute a single job. But real pipelines have multiple stages. Let’s add that.

Version 2: Multi-Stage Pipeline with Sequential Execution

Now let’s add support for stages. Jobs in different stages should run in order:

#!/usr/bin/env python3
"""
Pipeline Runner v2: Multi-stage support
Executes jobs stage by stage in sequential order
"""

import yaml
import subprocess
import sys
from pathlib import Path
from collections import defaultdict


class Job:
    """Represents a single pipeline job."""

    def __init__(self, name, config):
        self.name = name
        self.image = config.get('image', 'python:3.11')
        self.script = config.get('script', [])
        self.stage = config.get('stage', 'test')

    def __repr__(self):
        return f"Job({self.name}, stage={self.stage})"


class JobExecutor:
    """Executes a job in a Docker container."""

    def __init__(self, workspace):
        self.workspace = Path(workspace).resolve()

    def run(self, job):
        """Execute a job and stream output."""
        print(f"[{job.name}] Starting job...")

        script = ' && '.join(job.script)

        cmd = [
            'docker', 'run', '--rm',
            '-v', f'{self.workspace}:/workspace',
            '-w', '/workspace',
            job.image,
            'sh', '-c', script
        ]

        try:
            process = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
                text=True,
                bufsize=1
            )

            for line in process.stdout:
                print(f"[{job.name}] {line}", end='')

            process.wait()

            if process.returncode == 0:
                print(f"[{job.name}] ✓ Job completed successfully")
                return True
            else:
                print(f"[{job.name}] ✗ Job failed with exit code {process.returncode}")
                return False

        except Exception as e:
            print(f"[{job.name}] ✗ Error: {e}")
            return False


class Pipeline:
    """Represents a pipeline with stages and jobs."""

    def __init__(self, config_file):
        self.config_file = Path(config_file)
        self.config = self._load_config()
        self.stages = self.config.get('stages', ['test'])
        self.jobs = self._parse_jobs()

    def _load_config(self):
        """Load and parse YAML configuration."""
        with open(self.config_file) as f:
            return yaml.safe_load(f)

    def _parse_jobs(self):
        """Parse jobs from configuration."""
        jobs = []
        for job_name, job_config in self.config.items():
            if job_name != 'stages' and isinstance(job_config, dict):
                jobs.append(Job(job_name, job_config))
        return jobs

    def _group_jobs_by_stage(self):
        """Group jobs by their stage."""
        stages = defaultdict(list)
        for job in self.jobs:
            stages[job.stage].append(job)
        return stages

    def run(self, workspace='.'):
        """Execute pipeline stage by stage."""
        print(f"\n{'='*60}")
        print(f"Starting pipeline: {self.config_file.name}")
        print(f"Stages: {' → '.join(self.stages)}")
        print(f"Total jobs: {len(self.jobs)}")
        print(f"{'='*60}\n")

        executor = JobExecutor(workspace)
        stages_with_jobs = self._group_jobs_by_stage()

        # Execute stages in order
        for stage in self.stages:
            stage_jobs = stages_with_jobs.get(stage, [])

            if not stage_jobs:
                continue

            print(f"\n{'─'*60}")
            print(f"Stage: {stage} ({len(stage_jobs)} job(s))")
            print(f"{'─'*60}\n")

            # Execute all jobs in this stage (sequentially for now)
            for job in stage_jobs:
                success = executor.run(job)
                if not success:
                    print(f"\n✗ Pipeline failed at stage '{stage}'")
                    return False

        print(f"\n{'='*60}")
        print("✓ Pipeline completed successfully!")
        print(f"{'='*60}\n")
        return True


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python runner_v2.py <pipeline.yml>")
        sys.exit(1)

    pipeline = Pipeline(sys.argv[1])
    success = pipeline.run()
    sys.exit(0 if success else 1)

Testing Version 2

Create a multi-stage pipeline:

# pipeline.yml
stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  image: python:3.11
  script:
    - echo "Building application..."
    - mkdir -p dist
    - echo "v1.0.0" > dist/version.txt

test-job:
  stage: test
  image: python:3.11
  script:
    - echo "Running tests..."
    - python -c "print('All tests passed!')"

deploy-job:
  stage: deploy
  image: alpine:latest
  script:
    - echo "Deploying application..."
    - cat dist/version.txt
    - echo "Deployment complete!"

Run it:

python runner_v2.py pipeline.yml

Now we have stages! But notice that if you have multiple jobs in the test stage, they still run sequentially. In real CI/CD, jobs in the same stage run in parallel. Let’s add that.

Version 3: Parallel Job Execution

Real pipeline runners execute jobs in the same stage in parallel. Let’s implement this using Python’s multiprocessing:

#!/usr/bin/env python3
"""
Pipeline Runner v3: Parallel execution
Executes jobs within a stage in parallel using multiprocessing
"""

import yaml
import subprocess
import sys
from pathlib import Path
from collections import defaultdict
from multiprocessing import Pool, Manager
from functools import partial


class Job:
    """Represents a single pipeline job."""

    def __init__(self, name, config):
        self.name = name
        self.image = config.get('image', 'python:3.11')
        self.script = config.get('script', [])
        self.stage = config.get('stage', 'test')

    def __repr__(self):
        return f"Job({self.name}, stage={self.stage})"


class JobExecutor:
    """Executes a job in a Docker container."""

    def __init__(self, workspace):
        self.workspace = Path(workspace).resolve()

    def run(self, job, output_queue=None):
        """Execute a job and stream output."""

        def log(msg):
            """Log a message (to queue if available, else stdout)."""
            if output_queue:
                output_queue.put(msg)
            else:
                print(msg)

        log(f"[{job.name}] Starting job...")

        script = ' && '.join(job.script)

        cmd = [
            'docker', 'run', '--rm',
            '-v', f'{self.workspace}:/workspace',
            '-w', '/workspace',
            job.image,
            'sh', '-c', script
        ]

        try:
            process = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
                text=True,
                bufsize=1
            )

            for line in process.stdout:
                log(f"[{job.name}] {line.rstrip()}")

            process.wait()

            if process.returncode == 0:
                log(f"[{job.name}] ✓ Job completed successfully")
                return (job.name, True, None)
            else:
                error_msg = f"Exit code {process.returncode}"
                log(f"[{job.name}] ✗ Job failed: {error_msg}")
                return (job.name, False, error_msg)

        except Exception as e:
            error_msg = str(e)
            log(f"[{job.name}] ✗ Error: {error_msg}")
            return (job.name, False, error_msg)


def run_job_parallel(job, workspace, output_queue):
    """Helper function for parallel execution."""
    executor = JobExecutor(workspace)
    return executor.run(job, output_queue)


class Pipeline:
    """Represents a pipeline with stages and parallel job execution."""

    def __init__(self, config_file):
        self.config_file = Path(config_file)
        self.config = self._load_config()
        self.stages = self.config.get('stages', ['test'])
        self.jobs = self._parse_jobs()

    def _load_config(self):
        """Load and parse YAML configuration."""
        with open(self.config_file) as f:
            return yaml.safe_load(f)

    def _parse_jobs(self):
        """Parse jobs from configuration."""
        jobs = []
        for job_name, job_config in self.config.items():
            if job_name != 'stages' and isinstance(job_config, dict):
                jobs.append(Job(job_name, job_config))
        return jobs

    def _group_jobs_by_stage(self):
        """Group jobs by their stage."""
        stages = defaultdict(list)
        for job in self.jobs:
            stages[job.stage].append(job)
        return stages

    def run(self, workspace='.'):
        """Execute pipeline with parallel job execution per stage."""
        print(f"\n{'='*60}")
        print(f"Starting pipeline: {self.config_file.name}")
        print(f"Stages: {' → '.join(self.stages)}")
        print(f"Total jobs: {len(self.jobs)}")
        print(f"{'='*60}\n")

        workspace = Path(workspace).resolve()
        stages_with_jobs = self._group_jobs_by_stage()

        # Execute stages in order
        for stage in self.stages:
            stage_jobs = stages_with_jobs.get(stage, [])

            if not stage_jobs:
                continue

            print(f"\n{'─'*60}")
            print(f"Stage: {stage} ({len(stage_jobs)} job(s))")
            print(f"{'─'*60}\n")

            if len(stage_jobs) == 1:
                # Single job - run directly
                executor = JobExecutor(workspace)
                job_name, success, error = executor.run(stage_jobs[0])
                if not success:
                    print(f"\n✗ Pipeline failed at stage '{stage}'")
                    return False
            else:
                # Multiple jobs - run in parallel
                manager = Manager()
                output_queue = manager.Queue()

                # Create a partial function with workspace and queue
                run_func = partial(run_job_parallel, workspace=workspace, output_queue=output_queue)

                # Execute jobs in parallel
                with Pool(processes=len(stage_jobs)) as pool:
                    # Start async execution
                    results = pool.map_async(run_func, stage_jobs)

                    # Print output as it arrives
                    while True:
                        if results.ready() and output_queue.empty():
                            break

                        if not output_queue.empty():
                            print(output_queue.get())

                    # Get results
                    job_results = results.get()

                    # Check if all jobs succeeded
                    if not all(success for _, success, _ in job_results):
                        failed_jobs = [name for name, success, _ in job_results if not success]
                        print(f"\n✗ Pipeline failed at stage '{stage}'")
                        print(f"  Failed jobs: {', '.join(failed_jobs)}")
                        return False

        print(f"\n{'='*60}")
        print("✓ Pipeline completed successfully!")
        print(f"{'='*60}\n")
        return True


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python runner_v3.py <pipeline.yml>")
        sys.exit(1)

    pipeline = Pipeline(sys.argv[1])
    success = pipeline.run()
    sys.exit(0 if success else 1)

Testing Version 3

Create a pipeline with parallel jobs:

# pipeline.yml
stages:
  - build
  - test

build-job:
  stage: build
  image: python:3.11
  script:
    - echo "Building..."
    - sleep 2

unit-tests:
  stage: test
  image: python:3.11
  script:
    - echo "Running unit tests..."
    - sleep 3
    - echo "Unit tests passed!"

integration-tests:
  stage: test
  image: python:3.11
  script:
    - echo "Running integration tests..."
    - sleep 3
    - echo "Integration tests passed!"

Run it and notice that unit-tests and integration-tests run simultaneously!

Now we have parallel execution, but there’s a problem: what if test jobs need artifacts from the build job? We need dependency management and artifact passing.

Version 4: Dependencies and Artifacts

This is where it gets interesting. We need to:

1. Build a dependency graph (which jobs need which other jobs)
2. Execute jobs in topological order (respecting dependencies)
3. Pass artifacts between jobs

#!/usr/bin/env python3
"""
Pipeline Runner v4: Dependencies and Artifacts
Supports job dependencies, topological sorting, and artifact passing
"""

import yaml
import subprocess
import sys
import shutil
from pathlib import Path
from collections import defaultdict, deque
from multiprocessing import Pool, Manager
from functools import partial


class Job:
    """Represents a single pipeline job."""

    def __init__(self, name, config):
        self.name = name
        self.image = config.get('image', 'python:3.11')
        self.script = config.get('script', [])
        self.stage = config.get('stage', 'test')
        self.artifacts = config.get('artifacts', {}).get('paths', [])
        self.needs = config.get('needs', [])  # Job dependencies

    def __repr__(self):
        return f"Job({self.name}, stage={self.stage})"


class ArtifactManager:
    """Manages artifact storage and retrieval."""

    def __init__(self, workspace):
        self.workspace = Path(workspace).resolve()
        self.artifact_dir = self.workspace / '.pipeline_artifacts'
        self.artifact_dir.mkdir(exist_ok=True)

    def save_artifacts(self, job_name, artifact_paths):
        """Save artifacts from a job."""
        if not artifact_paths:
            return

        job_artifact_dir = self.artifact_dir / job_name
        job_artifact_dir.mkdir(exist_ok=True)

        for artifact_path in artifact_paths:
            src = self.workspace / artifact_path
            if src.exists():
                dst = job_artifact_dir / artifact_path
                dst.parent.mkdir(parents=True, exist_ok=True)

                if src.is_dir():
                    shutil.copytree(src, dst, dirs_exist_ok=True)
                else:
                    shutil.copy2(src, dst)

                print(f"  Saved artifact: {artifact_path}")

    def load_artifacts(self, job_names):
        """Load artifacts from dependent jobs."""
        for job_name in job_names:
            job_artifact_dir = self.artifact_dir / job_name
            if not job_artifact_dir.exists():
                continue

            # Copy artifacts to workspace
            for item in job_artifact_dir.rglob('*'):
                if item.is_file():
                    rel_path = item.relative_to(job_artifact_dir)
                    dst = self.workspace / rel_path
                    dst.parent.mkdir(parents=True, exist_ok=True)
                    shutil.copy2(item, dst)

    def cleanup(self):
        """Remove all artifacts."""
        if self.artifact_dir.exists():
            shutil.rmtree(self.artifact_dir)


class JobExecutor:
    """Executes a job in a Docker container."""

    def __init__(self, workspace, artifact_manager):
        self.workspace = Path(workspace).resolve()
        self.artifact_manager = artifact_manager

    def run(self, job, output_queue=None):
        """Execute a job and stream output."""

        def log(msg):
            if output_queue:
                output_queue.put(msg)
            else:
                print(msg)

        log(f"[{job.name}] Starting job...")

        # Load artifacts from dependencies
        if job.needs:
            log(f"[{job.name}] Loading artifacts from: {', '.join(job.needs)}")
            self.artifact_manager.load_artifacts(job.needs)

        script = ' && '.join(job.script)

        cmd = [
            'docker', 'run', '--rm',
            '-v', f'{self.workspace}:/workspace',
            '-w', '/workspace',
            job.image,
            'sh', '-c', script
        ]

        try:
            process = subprocess.Popen(
                cmd,
                stdout=subprocess.PIPE,
                stderr=subprocess.STDOUT,
                text=True,
                bufsize=1
            )

            for line in process.stdout:
                log(f"[{job.name}] {line.rstrip()}")

            process.wait()

            if process.returncode == 0:
                # Save artifacts
                if job.artifacts:
                    log(f"[{job.name}] Saving artifacts...")
                    self.artifact_manager.save_artifacts(job.name, job.artifacts)

                log(f"[{job.name}] ✓ Job completed successfully")
                return (job.name, True, None)
            else:
                error_msg = f"Exit code {process.returncode}"
                log(f"[{job.name}] ✗ Job failed: {error_msg}")
                return (job.name, False, error_msg)

        except Exception as e:
            error_msg = str(e)
            log(f"[{job.name}] ✗ Error: {error_msg}")
            return (job.name, False, error_msg)


def run_job_parallel(job, workspace, artifact_manager, output_queue):
    """Helper function for parallel execution."""
    executor = JobExecutor(workspace, artifact_manager)
    return executor.run(job, output_queue)


class Pipeline:
    """Represents a pipeline with dependency-aware execution."""

    def __init__(self, config_file):
        self.config_file = Path(config_file)
        self.config = self._load_config()
        self.stages = self.config.get('stages', ['test'])
        self.jobs = self._parse_jobs()

    def _load_config(self):
        """Load and parse YAML configuration."""
        with open(self.config_file) as f:
            return yaml.safe_load(f)

    def _parse_jobs(self):
        """Parse jobs from configuration."""
        jobs = []
        for job_name, job_config in self.config.items():
            if job_name != 'stages' and isinstance(job_config, dict):
                jobs.append(Job(job_name, job_config))
        return jobs

    def _topological_sort(self, jobs):
        """
        Sort jobs in topological order based on dependencies.
        Returns list of job groups where each group can run in parallel.
        """
        # Build adjacency list and in-degree count
        job_map = {job.name: job for job in jobs}
        in_degree = {job.name: 0 for job in jobs}
        adjacency = defaultdict(list)

        for job in jobs:
            for dep in job.needs:
                if dep in job_map:
                    adjacency[dep].append(job.name)
                    in_degree[job.name] += 1

        # Find jobs with no dependencies
        queue = deque([name for name, degree in in_degree.items() if degree == 0])
        execution_order = []

        while queue:
            # All jobs in current queue can run in parallel
            current_batch = list(queue)
            execution_order.append([job_map[name] for name in current_batch])

            queue.clear()

            # Process current batch
            for job_name in current_batch:
                for dependent in adjacency[job_name]:
                    in_degree[dependent] -= 1
                    if in_degree[dependent] == 0:
                        queue.append(dependent)

        # Check for cycles
        if sum(len(batch) for batch in execution_order) != len(jobs):
            raise ValueError("Circular dependency detected in job dependencies")

        return execution_order

    def _group_jobs_by_stage(self):
        """Group jobs by their stage."""
        stages = defaultdict(list)
        for job in self.jobs:
            stages[job.stage].append(job)
        return stages

    def _execute_job_batch(self, jobs, workspace, artifact_manager):
        """Execute a batch of jobs in parallel."""
        if len(jobs) == 1:
            # Single job - run directly
            executor = JobExecutor(workspace, artifact_manager)
            job_name, success, error = executor.run(jobs[0])
            return [(job_name, success, error)]
        else:
            # Multiple jobs - run in parallel
            manager = Manager()
            output_queue = manager.Queue()

            # Create a partial function
            run_func = partial(
                run_job_parallel,
                workspace=workspace,
                artifact_manager=artifact_manager,
                output_queue=output_queue
            )

            # Execute jobs in parallel
            with Pool(processes=len(jobs)) as pool:
                results = pool.map_async(run_func, jobs)

                # Print output as it arrives
                while True:
                    if results.ready() and output_queue.empty():
                        break

                    if not output_queue.empty():
                        print(output_queue.get())

                return results.get()

    def run(self, workspace='.'):
        """Execute pipeline with dependency resolution."""
        print(f"\n{'='*60}")
        print(f"Starting pipeline: {self.config_file.name}")
        print(f"Stages: {' → '.join(self.stages)}")
        print(f"Total jobs: {len(self.jobs)}")
        print(f"{'='*60}\n")

        workspace = Path(workspace).resolve()
        artifact_manager = ArtifactManager(workspace)
        stages_with_jobs = self._group_jobs_by_stage()

        try:
            # Execute stages in order
            for stage in self.stages:
                stage_jobs = stages_with_jobs.get(stage, [])

                if not stage_jobs:
                    continue

                print(f"\n{'─'*60}")
                print(f"Stage: {stage} ({len(stage_jobs)} job(s))")
                print(f"{'─'*60}\n")

                # Sort jobs by dependencies
                try:
                    execution_batches = self._topological_sort(stage_jobs)
                except ValueError as e:
                    print(f"✗ Error: {e}")
                    return False

                # Execute batches in order
                for batch in execution_batches:
                    job_results = self._execute_job_batch(batch, workspace, artifact_manager)

                    # Check if all jobs succeeded
                    if not all(success for _, success, _ in job_results):
                        failed_jobs = [name for name, success, _ in job_results if not success]
                        print(f"\n✗ Pipeline failed at stage '{stage}'")
                        print(f"  Failed jobs: {', '.join(failed_jobs)}")
                        return False

            print(f"\n{'='*60}")
            print("✓ Pipeline completed successfully!")
            print(f"{'='*60}\n")
            return True

        finally:
            # Cleanup artifacts
            artifact_manager.cleanup()


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python runner_v4.py <pipeline.yml>")
        sys.exit(1)

    pipeline = Pipeline(sys.argv[1])
    success = pipeline.run()
    sys.exit(0 if success else 1)