A new task appears in PENDING. It changes to RUNNING, but the old task does not go away. A minute later the old one starts draining. Eventually it disappears and the service returns to steady state. If you only watch the task count, the whole thing looks oddly slow and a little random.

It is neither. Four systems are working on the deployment at the same time:

• The ECS scheduler is enforcing minimum and maximum task counts.
• The ECS agent or Fargate runtime is provisioning the task and starting its containers.
• The load balancer is deciding whether the new target should receive traffic.
• The old container is trying to finish requests before ECS kills it.

Most of the confusing failures I see come from treating those four systems as one thing. A task can be running but not ready. It can be healthy in ECS but unhealthy in the target group. It can be removed from the load balancer and still spend another minute shutting down.

We are going to follow one deployment from the API call to the final stopped task. At each pause, we will look at which system owns the wait and what condition it needs before the rollout can continue.

The example service

We will use a small Fargate service behind an Application Load Balancer:

service: api
desired count: 4
current task definition: api:41
new task definition: api:42
minimumHealthyPercent: 100
maximumPercent: 200
healthCheckGracePeriodSeconds: 60

The service begins with four healthy tasks running revision api:41. Our pipeline registers api:42, then calls UpdateService to point the service at the new revision.

The deployment settings create a range:

minimum healthy = ceil(4 × 100%) = 4
maximum running or pending = floor(4 × 200%) = 8

ECS must keep at least four healthy tasks available, but it may temporarily run as many as eight. With enough Fargate or EC2 capacity, the scheduler can start all four replacements before touching the old tasks.

When a rollout behaves strangely, I calculate this range before looking anywhere else.

The service revision changes first

Registering a task definition does not deploy it. It only creates an immutable blueprint such as api:42.

Updating the service is what starts the deployment:

aws ecs update-service \
  --cluster production \
  --service api \
  --task-definition api:42

ECS records the configuration it is leaving and the configuration it is trying to reach. In the current ECS deployment model, these are the source and target service revisions. A service revision includes more than the task definition: it records the workload configuration ECS is attempting to deploy and gives rollback a known previous state.

No customer traffic has moved yet. ECS has only changed desired state:

source: api:41, four healthy tasks
target: api:42, zero healthy tasks
deployment: IN_PROGRESS

The service scheduler now begins reconciling reality with the new declaration.

The scheduler finds room for the new tasks

The scheduler does not blindly replace one task at a time. It works inside the range created by minimumHealthyPercent and maximumPercent.

minimumHealthyPercent is the availability floor. ECS rounds it up.

maximumPercent is the concurrency ceiling for tasks in RUNNING or PENDING. ECS rounds it down.

The same four-task service behaves quite differently as those settings change:

These percentages affect availability, but on ECS with EC2 they also decide how much spare capacity a deployment needs. A service configured for 100/200 may run the old and new revisions together. If the cluster is already full, those new tasks have nowhere to go. Fargate hides the hosts, but account quotas, IP availability, and platform capacity can still block placement.

There is also a rounding trap. With a desired count of three and maximumPercent: 125, the upper limit is:

floor(3 × 1.25) = 3

ECS cannot start a fourth task. If minimumHealthyPercent also prevents it from stopping an old task, the deployment has no legal move. ECS emits a service event telling you that the deployment configuration cannot start or stop a task.

The first task pins the image

Task definition api:42 may say:

{
  "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/api:production"
}

That tag is mutable. It can point to a different image tomorrow.

By default, ECS resolves image tags to digests during a deployment so that every task in the service runs identical image content. For a service with multiple tasks, the first new task is used to establish the image digest, and the remaining tasks use that digest.

Overwriting a tag does not change a running task. ECS needs a new deployment before it resolves and launches the new image. Unique tags, preferably the Git SHA, also save a lot of guesswork during an incident.

api:production                         mutable pointer
api@sha256:8d6c...                     actual image content

The task then moves through the early lifecycle:

PROVISIONING → PENDING → ACTIVATING → RUNNING

Depending on the launch type and configuration, this includes finding capacity, attaching an ENI, pulling images, creating containers, configuring networking, registering targets, and starting essential containers.

RUNNING is one of the most misleading words in the ECS console. It means the containers are running. It does not mean the application is ready for customer traffic, and ECS may not count the task as healthy yet.

RUNNING is followed by another wait

A task can be evaluated by two separate health systems:

1. A container health check defined in the ECS task definition.
2. A load-balancer target group health check.

If an essential container has an ECS health check and the service uses a load balancer, both must pass before the scheduler counts the task as healthy for the deployment.

A container health check runs inside the container:

{
  "healthCheck": {
    "command": [
      "CMD-SHELL",
      "curl -f http://localhost:8080/health || exit 1"
    ],
    "interval": 10,
    "timeout": 5,
    "retries": 3,
    "startPeriod": 30
  }
}

The startPeriod belongs to the container health check. Failed checks during that bootstrap window do not count toward the retry limit. If a check succeeds during the start period, the container becomes healthy and later failures count normally.

healthCheckGracePeriodSeconds is different. It belongs to the ECS service and tells the service scheduler to ignore unhealthy container, load-balancer, or VPC Lattice health status for a period after each task starts.

The two timers sound interchangeable, but they sit at different layers:

container startPeriod
  protects the container health-check retry counter during startup

service health-check grace period
  prevents the ECS scheduler from replacing a new task during startup

Neither setting sends traffic to an unhealthy target. The Application Load Balancer still follows its own target health state. The grace period only changes how the ECS scheduler reacts to an unhealthy result.

For a newly registered ALB target, one successful health check is enough to mark it healthy. The target group’s healthy-threshold count applies when a previously unhealthy target is recovering. With the default 30-second health-check interval, even a healthy application may spend noticeable time waiting for the next probe.

Before the first api:42 task counts toward the rollout, all of this must be true:

ECS task state: RUNNING
container health: HEALTHY
ALB target health: healthy
counts toward deployment minimum: yes
receiving customer traffic: yes

Now the new revision has proved that it can serve traffic. Until this point, stopping an old task would spend availability on a replacement that had not earned it.

New tasks come in, old tasks drain

As new tasks become healthy, the scheduler gains room to remove old ones without crossing the availability floor.

With our 100/200 settings, the rollout may look like this:

time     api:41 healthy     api:42 starting/healthy     total
t0              4                     0                   4
t1              4                     4 starting          8
t2              4                     4 healthy           8
t3              0                     4 healthy           4

Do not read that table as a promise that all four tasks move together. The scheduler chooses the batches. Placement capacity, startup time, health results, and throttling can turn the same configuration into a more incremental rollout.

When ECS decides to stop an old task behind an ALB, the task does not jump directly from RUNNING to STOPPED.

It moves through the shutdown side of the lifecycle:

RUNNING → DEACTIVATING → STOPPING → DEPROVISIONING → STOPPED

During DEACTIVATING, ECS deregisters the task from the target group. The target enters draining, and the load balancer stops assigning it new requests while allowing existing connections time to complete according to the target group’s deregistration delay.

Then ECS stops the containers. On Linux, the container receives the signal defined by its image STOPSIGNAL, which is SIGTERM by default. ECS waits for the container’s stopTimeout. If the process is still alive after that window, it receives SIGKILL.

For graceful shutdown, all three layers need to agree:

ALB deregistration delay
    ≥ longest request or connection you intend to preserve

ECS stopTimeout
    ≥ time the application needs after SIGTERM

application shutdown handler
    stops accepting work and exits before stopTimeout

If the application ignores SIGTERM, no ECS setting can make its shutdown graceful. ECS will eventually kill it.

Long-lived WebSocket connections, streaming responses, background jobs, and queue consumers require special attention. A web server can stop accepting new requests and finish active ones. A worker may need to stop polling, return an in-flight message to the queue, or extend its visibility timeout. Rolling deployment safety is partly an application concern.

ECS reaches steady state

The deployment completes after the target revision reaches the desired count and the old revision no longer has active tasks in the rollout.

For our service:

api:41: 0 running tasks
api:42: 4 running and healthy tasks
desired count: 4
deployment: SUCCESSFUL

Your CI system does not perform the rollout. A deployment action usually registers the task definition, updates the service and, if configured, polls ECS until the service stabilizes. That is what the official GitHub action does when wait-for-service-stability is set to true. The scheduler still runs the deployment.

So when a pipeline appears frozen, the useful evidence is often somewhere else. ECS may be waiting for a health check, placement capacity, a draining connection, or the circuit-breaker threshold.

Start with ECS service events and deployment details, not the CI runner logs.

Useful commands include:

aws ecs describe-services \
  --cluster production \
  --services api

aws ecs list-service-deployments \
  --cluster production \
  --service api

aws ecs describe-service-deployments \
  --service-deployment-arns SERVICE_DEPLOYMENT_ARN

aws ecs list-tasks \
  --cluster production \
  --service-name api

Service deployment history includes the source and target revisions, deployment state, failed-task count, alarm state, timestamps, and rollback details. ECS retains recent deployment history, which is much more useful than reconstructing a rollout from a few console event messages.

What happens when the new tasks fail?

Failed deployments usually split into two groups. The difference tells you where to start looking.

The task never reaches RUNNING

The task may have no capacity, fail to pull its image, or start with an execution role that cannot retrieve a secret. ENI attachment failures also happen here. So does an essential container that exits during startup.

These are launch failures. In its first stage, the deployment circuit breaker counts consecutive tasks that fail to reach RUNNING.

The task runs but never becomes healthy

Sometimes the process starts but listens on the wrong port, or the health endpoint returns a failing status. The ALB security group may not reach the task. A slow application can run past its grace period. I have also seen health checks fail because the image did not contain curl, even though the application itself was fine.

These are health failures. After at least one new task reaches RUNNING, the circuit breaker moves to its second stage and watches container, load-balancer, and service-discovery health.

The circuit breaker threshold is based on half the desired count, bounded to a minimum of 3 and a maximum of 200:

threshold = ceil(0.5 × desired count), bounded to [3, 200]

For a service with a desired count of one, the threshold is still three. A tiny service may launch the same broken task several times before ECS finally calls the deployment failed.

Enable rollback explicitly:

{
  "deploymentCircuitBreaker": {
    "enable": true,
    "rollback": true
  },
  "minimumHealthyPercent": 100,
  "maximumPercent": 200
}

When the deployment fails, ECS can roll back to the last service revision that completed successfully. CloudWatch alarms cover a different failure mode: the tasks are technically healthy, but latency or error rate gets worse after the release.

I use both because they answer different questions:

circuit breaker
  catches tasks that cannot launch or become healthy

CloudWatch deployment alarms
  catch applications that are running but behaving badly

A /health endpoint returning 200 cannot tell you that checkout latency tripled or every database write is failing. The scheduler only knows what you expose to it.

Why healthy deployments still cause errors

ECS can execute a perfect rolling deployment and users can still see failures.

The new and old versions are incompatible

During a rolling deployment, both versions receive traffic. A destructive database migration, incompatible queue payload, or changed cache format can break one revision while the other is still alive.

Database changes should generally follow expand-and-contract:

1. Add the new schema while keeping the old schema valid.
2. Deploy code that can work with both representations.
3. Migrate data.
4. Remove the old schema in a later deployment.

Readiness is too shallow

Returning 200 because the HTTP process started is not enough if the application cannot reach a dependency required for serving requests. But checking every downstream system can also cause a cascading failure by removing all targets during a shared dependency outage.

The useful question is narrower: what condition means this particular task should stop receiving traffic? Write the health check around that contract.

Shutdown is not graceful

If the process exits immediately on SIGTERM, active requests die even though the ALB is draining the target. If it waits longer than stopTimeout, ECS kills it anyway.

Sticky sessions or local state hide the overlap

Rolling deployments assume tasks are replaceable. Sessions stored only in process memory, local uploads, or singleton background work make replacement unsafe. The scheduler cannot protect state it does not know exists.

The deployment settings I start with

For a normal stateless HTTP service on Fargate behind an ALB, I start here:

desired count: at least 2
minimumHealthyPercent: 100
maximumPercent: 200
deployment circuit breaker: enabled with rollback
CloudWatch alarms: 5xx rate and latency, with rollback
health-check grace period: measured startup time plus margin
image tag: immutable Git SHA
container stopTimeout: matched to graceful shutdown behavior

I keep the ALB health check cheap and local. I test SIGTERM handling outside production. I also alarm on SERVICE_DEPLOYMENT_FAILED through EventBridge because an automatic rollback that nobody notices is still a failed release.

If doubling task count is too expensive or the EC2 cluster cannot hold it, lower maximumPercent carefully and verify that the scheduler still has room to make progress. Saving temporary capacity is not useful if it creates a deployment deadlock or reduces availability below what the application can tolerate.

A practical debugging order

When a deployment is slow or stuck, inspect it in this order:

1. Deployment state: What are the source and target service revisions?
2. Scheduler math: Do the minimum and maximum percentages allow a task to start or stop?
3. Service events: Is placement, capacity, IAM, image pulling, or networking failing?
4. Task lifecycle: Are new tasks stuck in PENDING, stopping before RUNNING, or running and then replaced?
5. Stopped-task reason: What did ECS report for the failed task and essential container?
6. Target health: Is the ALB target initial, unhealthy, healthy, or draining? What reason does the target group report?
7. Application logs: Did the process bind the expected port and finish startup?
8. Timers: Are the container start period, service grace period, ALB interval, deregistration delay, and stop timeout consistent?
9. Rollback controls: Is the circuit breaker enabled, and are application alarms attached?

The order matters. There is no point staring at application logs if ECS never started the container. Likewise, changing a health endpoint will not fix a target group that cannot reach the task through its security group.

What to remember

An ECS rolling deployment is a constrained reconciliation loop:

declare a new service revision
        ↓
calculate the legal task-count range
        ↓
start tasks from the new revision
        ↓
prove they can run and receive traffic
        ↓
drain traffic from the old tasks
        ↓
give old processes time to exit
        ↓
stop the old revision and reach steady state

The scheduler protects task counts. Health checks decide which tasks receive traffic. Draining gives active requests time to finish. The circuit breaker catches a rollout that cannot converge. CloudWatch alarms catch the more annoying case, where the rollout converges and the application still gets worse.

When the ECS console sits on Deployment in progress, I check three things: the task state, the target state, and whichever timer is active. So far, the pause has always been hiding in one of them.

Sources and further reading

• Deploy Amazon ECS services by replacing tasks
• Amazon ECS task lifecycle
• How the ECS deployment circuit breaker detects failures
• Amazon ECS service deployment history
• Amazon ECS service revisions
• Optimize load-balancer health checks for ECS
• Determine task health with container health checks
• Automate rollback with CloudWatch alarms

I’ve written before about the ECS decisions that waste six weeks. This post is the prequel: what ECS actually is, how it maps onto the Kubernetes concepts you already know, and what you stop carrying on your pager when you choose it. There are a few interactive diagrams below. Click around in them; they teach the model faster than prose does.

One thing before we start: this is not a “Kubernetes bad” post. EKS is the right choice for some teams, and I’ll tell you exactly which ones at the end. But I’ve watched too many three-person teams default to EKS because it felt like the serious choice, without anyone explaining what they were signing up to operate.

ECS is an orchestrator. That’s it.

Strip away the branding and every container orchestrator does the same job: you declare what should be running, and a control loop makes reality match the declaration. Kubernetes does this. Nomad does this. ECS does this.

ECS just exposes far fewer moving parts to you. Here’s the whole object model. Click each piece:

If you know Kubernetes, the translation table is short enough to memorize over coffee:

That last column is where the story actually lives. In Kubernetes, “where compute comes from” and “how pods get IPs” and “how workloads get cloud credentials” are all decisions with an ecosystem of competing answers. In ECS they’re defaults. You don’t pick a CNI. You don’t install an IRSA webhook. There’s one way, it’s boring, and it works.

The reconciliation loop — same idea, fewer layers

The core idea both systems share: you declare desired state, a control loop enforces it. This is the part I find people understand instantly once they watch it instead of reading about it.

Below is an ECS service with desired count: 4. Click a task to kill it, then watch the scheduler notice and replace it. Then hit deploy and watch a rolling deployment do exactly what a Kubernetes Deployment rollout does: bring up new tasks, drain old ones, never drop below healthy.

That’s a Deployment rollout and a ReplicaSet self-heal, except nobody installed anything to get it. There’s no controller manager to version. You get all of this the moment you create a service.

When I help teams ship on ECS, this is where it clicks: you already understand ECS. If you can reason about desired state and reconciliation, the orchestration knowledge transfers completely. What doesn’t transfer is the operational surface area, and that’s the actual argument.

What you stop operating

This is the comparison that matters for a small team, and it’s the one nobody draws. The question isn’t which scheduler is smarter. They’re both fine. The question is whose pager each layer lands on.

Toggle ECS between Fargate and EC2 to see the middle ground:

Look at the EKS column. Six of the seven layers are yours. None of them are your product.

The upgrade treadmill deserves special attention because it’s the one that quietly eats small teams. Kubernetes ships about three releases a year, and EKS standard support for each lands around 14 months. That means a recurring, unskippable project roughly once a year, forever: test the control plane upgrade, upgrade the add-ons in the right order, chase whatever deprecated APIs your manifests use, then roll the nodes. Skip it and AWS moves you to extended support at six times the control plane price. For a platform team of 15, that’s Tuesday. For a team of four, it’s a sprint per year spent running to stand still. And there’s a quieter cost on top: you have to stay the kind of team that can do this safely.

ECS doesn’t have a version. I want to make sure that lands. There is no upgrade, no deprecation cycle, no “v1.29 removes the API your ALB controller depends on.” The control plane changed under you a hundred times last year and you never noticed. I have ECS services from 2021 that have never needed a maintenance commit. Infrastructure that doesn’t generate homework is worth more to a small team than anything on the Kubernetes feature list. It’s the same reason I tell teams to pick boring options everywhere else in the stack: boring means you debug your app, not your platform.

On raw cost, the EKS control plane is about $73 a month per cluster and ECS’s is free, and that’s the least interesting line in the comparison. Run the numbers on engineering time instead. One sprint of one engineer’s time per year on cluster maintenance is $10-20k. The biggest AWS savings I’ve ever found came from deleting complexity, not from rightsizing it.

What you give up

If this were one-sided, EKS wouldn’t exist. Here’s what you actually lose.

The big one is the operator ecosystem. Kubernetes has operators for Postgres, Kafka, cert-manager, external-dns, ArgoCD, all debugged by thousands of teams over a decade. ECS has no CRDs and no operator pattern. The AWS answer is “use the managed service”: RDS instead of a Postgres operator, MSK instead of Strimzi. That works right up until you need something AWS doesn’t sell.

Tooling in general follows the same line. Vendors ship a Helm chart, not a task definition. Kustomize, the CNCF landscape, none of it targets ECS. And your deployment layer is AWS-native, so a future move off AWS means rewriting it. Your containers move unchanged, but the wiring around them doesn’t.

There’s also the hiring thing, and I won’t pretend it isn’t real. Engineers want Kubernetes on their CV. ECS knowledge is real orchestration knowledge and the concepts transfer completely, as the diagrams above show, but nobody’s career was ever advanced by the phrase “task definition.”

And ECS has a control ceiling. Custom schedulers, topology spread, network policy, the more exotic probe and init semantics: Kubernetes gives you knobs ECS simply doesn’t have. Most web products never touch them. If yours genuinely does, you’ll feel the ceiling and you’ll resent it.

So when is EKS the right call?

EKS earns its keep when at least one of these is true:

• Someone owns the platform. You have, or are hiring, people whose actual job is cluster operations, so the pager layers above land on a team that exists.
• You’re running stateful infrastructure on-cluster that AWS doesn’t offer as a managed service, and you need the operator ecosystem for it.
• Multi-cloud or on-prem is a real requirement: contractual, regulatory, or your customers deploy your software into their clusters.
• Your team is already fluent. K8s veterans ship faster on EKS than they would learning anything else. The tax is only a tax if you haven’t already paid it.

If none of those describe you, and for most sub-ten-engineer teams shipping a web product none do, then Kubernetes isn’t buying you capability. It’s buying you a second job.

The takeaway

ECS is not “Kubernetes for beginners.” It’s the same control loop idea with a deliberately smaller operational surface. Same desired state, same reconciliation, same rolling deploys, minus the version treadmill, the add-on stack, and the node fleet. You’ve seen the whole object model in this post. There is no part two where the hidden complexity lives.

Small teams don’t lose because they picked the wrong orchestrator. They lose because their best engineers spent the year operating infrastructure the product didn’t need. Pick the tool that generates the least homework, ship, and revisit when you have the head count to afford opinions.

If you’re starting an ECS build-out, the companion post on the 5 ECS decisions that waste 6 weeks covers the concrete choices: Fargate vs EC2, service discovery, CI/CD, secrets, and monitoring.

If this post saved you a meeting, it did its job. I write about AWS, DevOps, and building things from scratch. Subscribe via RSS, or find me on Twitter.

They are both right, which is the problem. The number on the screen is real and it is also not the number you actually wait for. And the format you should pick was never really about that number anyway.

I have gone through this decision enough times now, on my own machine and for clients standing up local inference, that I want to write down the part nobody puts in the comparison tables: GGUF versus MLX is a five-question decision, and only one of those questions is about speed.

What you are actually choosing between

GGUF is the file format from the llama.cpp project. One file holds the quantized weights, the tokenizer, the chat template, and the metadata, and any runtime that can load it will run the model. That includes llama.cpp itself, Ollama, LM Studio, KoboldCPP, and a handful of others. It runs on basically everything: CPU, NVIDIA, AMD, Apple Metal, even a Raspberry Pi if you are patient. Portability is the whole point of the format.

MLX is not a file format. It is Apple’s array framework, the rough equivalent of PyTorch built specifically for Apple Silicon. An MLX model is a directory of safetensors files plus a config that the runtime reads directly. You convert and quantize a model in one command with mlx_lm.convert. The catch is in the name: MLX runs on Apple Silicon and nowhere else.

One thing worth clearing up before we go further, because it shows up in half the comparisons and it is out of date: people say GGUF does clever mixed-precision quantization while MLX is stuck on flat uniform 4-bit. The first half is true. The second half is not. Apple walked through per-layer mixed precision in their WWDC25 session on running large language models with MLX, including the trick of keeping the embedding and output layers at 6-bit while the rest of the model sits at 4-bit. MLX can do it. It is just that most of the MLX builds floating around Hugging Face do not bother, so in practice you often are comparing GGUF’s mixed precision against a uniform MLX quant. Worth knowing when you read someone else’s quality benchmark.

The number on the screen is lying to you

Quick detour, because it poisons most of the benchmarks you will find.

The tokens-per-second figure your runtime prints while text is streaming measures decode speed, the rate at which the model emits new tokens. It does not include prefill, the time the model spends reading your prompt before it says anything. For a chatty exchange with a short prompt that does not matter much. For an agent that stuffs tool output, a chunk of a file, and a system prompt into every turn, prefill is most of what you wait for, and the streaming counter never sees it.

There is a benchmark writeup that made the rounds on r/LocalLLaMA where the author’s UI proudly reported nearly twice the tokens per second on MLX as on GGUF, and then the actual wall-clock time had GGUF finishing first on most of the real tasks. Same machine, same model. The counter was not wrong. It was just answering a different question than the one that mattered.

Keep that in your head for the whole rest of this post. When I say one format is “faster” below, I mean wall-clock on a real workload, not the number that scrolls past while tokens stream.

Five questions that actually decide it

1. How big is the model relative to your RAM?

This is the question that quietly settles a lot of arguments. Token generation is bounded by memory bandwidth, not compute. To emit one token the GPU has to read the entire model out of memory. On an M4 Pro with roughly 273 GB/s of bandwidth, a 4-bit 27B model weighing about 17 GB caps out near 16 tokens per second no matter what software you run. MLX cannot fetch bytes faster than the hardware allows, and neither can llama.cpp.

So for large models, the ones that fill most of your unified memory, the format barely matters for speed. They both hit the same wall. The interesting differences show up on smaller models, under roughly 8 to 14B, where the model fits comfortably and the bottleneck shifts from bandwidth to framework overhead. That is where MLX’s tighter, Apple-specific kernels pull ahead, often in the 15 to 40 percent range on single-user decode, and wider still on very small models that lean hardest on framework efficiency.

Small model, want it snappy: MLX has something real to offer. Big model that barely fits: pick on the other four questions, because speed is a wash.

2. Will this ever need to run somewhere other than a Mac?

If there is any chance the same artifact has to run on a Linux box, a cloud GPU, or a teammate’s non-Apple machine, you want GGUF. The same file moves between all of them. MLX does not leave Apple Silicon, full stop. If you ship MLX as your only build and then need a CUDA fallback, you are re-quantizing under pressure.

This one overrides almost everything else. Portability is not a performance feature, but it is the feature you miss most when it is gone.

3. What does your workload actually look like?

Not “what model,” but the shape of the traffic. Specifically the ratio of input to output.

Workloads that feed the model a lot and ask for a little (classification, tool-calling agents with short replies, RAG with a big injected context) lean toward GGUF. llama.cpp has more battle-tested prompt caching and FlashAttention, and MLX’s prefix caching has historically been the less reliable of the two, especially on newer hybrid-attention models. When prefill dominates the wall clock, that maturity wins.

Workloads that take a short prompt and generate a lot (summaries, long-form chat, brainstorming) lean toward MLX. Once the model is past prefill and just streaming tokens, MLX’s decode advantage compounds, and the longer the reply the more it pays off.

There is a crossover point that depends on both context size and reply length. With a small prompt, MLX needs a couple hundred output tokens before its faster decode makes up for slower prefill. With a few thousand tokens of context, it needs several hundred more. If your agent’s replies are 150 tokens and its context keeps growing, you are living on the wrong side of that crossover, and GGUF is the better call.

4. Do you want to train, or just run?

GGUF is an inference format. You download it, you run it, that is the relationship. If you want to fine-tune, you convert back to safetensors, find a GPU, do the work, and convert forward again.

MLX is a full framework. You can fine-tune with LoRA or QLoRA directly on the Mac, merge adapters, and run speculative decoding with a small draft model, all natively. If part of your reason for going local is to actually adapt models and not just serve them, MLX is the only serious option on Apple Silicon, and this question alone can decide the whole thing.

5. How much do you care about ecosystem and exact fit?

Two practical edges for GGUF here. First, coverage: every open model gets GGUF builds within hours of release, including the obscure ones. MLX coverage is good for popular models and lags for everything else. Second, granularity. GGUF gives you a long ladder of quant levels, Q4_K_M, Q5_K_M, Q6_K, the I-quants, and so on, so when you have exactly 16 GB to work with you can usually find a quant that fits. MLX builds are mostly published at 4-bit and 8-bit, so you sometimes get a 4-bit that is a hair too small for the quality you want and an 8-bit that will not fit.

The edge on MLX’s side: it tends to get support for new Apple hardware features first, because Apple ships the metal abstraction in MLX before llama.cpp catches up.

The flowchart

Put the five questions in order and most decisions fall out in about ten seconds.

• Need to run on anything other than Apple Silicon, now or later? → GGUF. Stop here, portability wins.
• Staying on Apple Silicon. Do you want to fine-tune or train on-device? → MLX.
• Inference only. Is your workload short-output and prefill-heavy (agents, RAG, classification)? → GGUF.
• Long outputs, interactive, single user, latency you can feel? → MLX.
• Need a precise quant to fit tight RAM, or running a just-released or obscure model? → GGUF.
• Still undecided? → GGUF. It is the conservative default. Ship it, and A/B an MLX build later if throughput becomes the constraint.

The short version: GGUF is what you pick when you are not sure, because it is the one that is hard to regret. MLX is what you pick when you own the hardware, run single-user, and have a specific reason, throughput on long outputs or on-device training, to want it.

Once you have picked, pick a quant level

The format is half the decision. The bit width is the other half, and the defaults are good but not always right.

Start at Q4_K_M for GGUF or 4-bit for MLX. Q4_K_M is the community default for a reason. It keeps most tensors at 4-bit, then bumps the quality-sensitive ones to 6-bit: the attention value weights and the feed-forward down-projection, on a portion of the layers. That holds quality better than a flat 4-bit quant at a small size cost. The reported quality loss against FP16 on MMLU is model-dependent but small: well under a point on a big model, creeping up toward a point or so on something under 8B, and a little more again for a uniform 4-bit MLX build. On a 30B-plus model that gap is noise. On something under 8B, especially on coding tasks where attention precision matters, it is visible, and you have two outs: stay on GGUF Q4_K_M, or move to MLX 6-bit, which closes the gap for roughly a 30 percent larger file.

If RAM is genuinely tight, GGUF’s I-quants with an importance matrix are the quality-per-byte champions at low bit widths. The cost is slower decode on CPU, so they make more sense when you are squeezing a model onto limited memory than when you are chasing speed.

One rule regardless of format: do not drop below roughly 3-bit without measuring quality on your own task. The aggregate benchmarks stop predicting what you will actually see down there.

Two traps that will flip your results

The bf16 trap on M1 and M2. A lot of MLX builds ship as bf16, and on the M1 and M2 that data type does not get the accelerated path that fp16 does. During prefill those weights run un-accelerated and the penalty multiplies across every input token, which is part of why some “MLX is slow” reports come from older hardware. The fix is a one-minute reconvert with --dtype float16. If you are on an M1 or M2 and MLX feels sluggish, check this before you blame the format.

Caching is the real variable. The biggest swings I have seen between runtimes were not about GGUF versus MLX at all, they were about whether prompt and KV caching actually worked for that model on that runtime. A runtime that reprocesses the full conversation every turn will lose to one that caches the prefix, regardless of format. Test caching with your real context lengths before you commit, and do not trust the streaming counter to tell you about it, because it never measures the part that caching fixes.

So which one

If you want the one-line version: GGUF is the conservative default, and you should reach for it whenever you are uncertain, need portability, or want a specific quant. Reach for MLX when you are locked to Apple Silicon, run single-user interactive workloads with long outputs, or want to fine-tune on the machine you already own.

And if you are choosing this for a team rather than a laptop, treat it as the architecture decision it is. The format you standardize on shapes your model coverage, your fallback options, and your serving setup for as long as the stack lives, and re-quantizing a fleet after the fact is the kind of avoidable week of work I keep getting hired to clean up. Decide it on the five questions, not on a screenshot.

I kept watching this happen. Ten minutes of exploration before any real work, every single time. If you work across multiple repos or come back to a project after a couple weeks, it’s worse. The agent is essentially reading the codebase for the first time, again.

I wanted to fix this.

The idea

A few weeks ago Karpathy tweeted about using LLMs to build personal knowledge bases. The workflow: collect raw sources, have an LLM compile them into a structured wiki of markdown files, then query and build on that wiki over time. Every query makes the wiki richer. The knowledge adds up.

The part that stuck with me: he’s not using fancy RAG. The LLM maintains its own index files and summaries, and at his scale (~100 articles, ~400K words) it just works. The LLM reads its own compiled knowledge to answer questions.

Codebases are raw data too. Source files are unstructured information that happens to be executable. What if the LLM compiled a codebase into a wiki the same way, with module overviews, architecture docs, concept articles, and then used that wiki as its starting point for every session?

That’s CodeWiki.

How it works

CodeWiki is a thin Rust CLI called cw paired with a Claude Code skill. The CLI handles git ops, directory scaffolding, and metadata. The agent does all the actual reading and writing. No API keys, no LLM calls from the CLI. Your agent is the intelligence.

When you run cw init in a repo, it creates a wiki directory at ~/.codewiki/<project>/ with this structure:

~/.codewiki/my-project/
├── _index.md         # master index
├── _architecture.md  # system overview
├── _patterns.md      # recurring patterns
├── _meta.yaml        # last compiled commit
├── modules/          # one article per module
├── concepts/         # cross-cutting concerns
├── decisions/        # why things are the way they are
├── learnings/        # bugs fixed, patterns discovered
└── queries/          # past Q&A, filed back

The first time you start a Claude Code session after init, the skill kicks in. The agent walks your codebase, reads the source files, and writes wiki articles. Module articles describe what each part of the code actually does. Not what it’s supposed to do, what it does. Key files, functions, data flow, connections to other modules.

Concept articles cut across modules. “How does error handling work across the system” or “how does data flow from request to response.” These are the questions that normally require reading eight files across four directories. The wiki answers them in one place.

Keeping it fresh

The wiki is only useful if it stays current. Every article has YAML frontmatter with a source_files field:

---
title: Authentication Module
type: module
source_files:
  - src/auth/middleware.py
  - src/auth/tokens.py
tags: [auth, middleware, jwt]
---

The CLI tracks which commit the wiki was last compiled against. When you start a new session, cw status diffs against that commit and cross-references changed files against every article’s source_files:

$ cw status
Changed since last compile (4964c23):
  M src/auth/middleware.py
  M src/auth/tokens.py

Stale articles:
  ! modules/auth.md

The agent sees this and knows exactly what to re-read and update. No guessing, no full recompile.

At session end, the agent writes learnings and decisions back into the wiki. Fixed a bug? That becomes learnings/auth-token-race-condition.md. Made a design decision? That’s decisions/switched-to-redis-sessions.md. Then it updates _meta.yaml with the current commit hash.

Next session picks up where this one left off.

The CLI

About 400 lines of Rust. Here are the commands:

cw init                # scaffold wiki for current repo
cw status              # what changed since last compile
cw path                # print wiki path
cw projects            # list all wikis
cw index               # rebuild _index.md from article frontmatter
cw meta update         # record current commit as compiled

cw setup claude-code   # install skill into Claude Code
cw setup codex         # install instructions for Codex
cw setup qmd           # register wiki as QMD search collection

The CLI doesn’t make any LLM calls. It handles the things agents are bad at: tracking git state, knowing which files changed, maintaining timestamps. The agent handles what it’s good at: reading code and writing about it.

Search with QMD

For larger wikis, QMD by Tobi Lutke adds proper search. It’s a local search engine for markdown with hybrid BM25 plus vector search plus a small reranker model. Running cw setup qmd registers your wiki as a searchable collection. The agent can then query the wiki through QMD’s MCP server during a session.

At the scale of most repos people actually work in, you probably don’t need it. A well organized wiki with an index file is enough for the LLM to navigate on its own. But when the wiki gets large, QMD keeps retrieval fast.

Viewing with Obsidian

All wiki articles live at ~/.codewiki/. Open that directory as an Obsidian vault and you get a browsable knowledge graph of all your projects. Articles use [[backlinks]] so modules connect to each other. The auth article links to [[database]] and [[api]]. You never have to write or edit these articles yourself. The agent maintains everything.

Why not RAG

Traditional RAG chunks your code, embeds it, retrieves fragments when you ask a question. You get decontextualized snippets and hope the LLM can stitch them together.

CodeWiki is different. The LLM reads the code and writes structured articles about it. The auth article already connects the middleware to the token service to the database layer. That connection doesn’t exist in any single source file. It exists in the compiled understanding.

Karpathy found the same thing with his research wiki. You don’t need vector search over raw data when you have a well organized collection of articles. The LLM reads the index, finds the relevant articles, reads those. Simple and it works.

Getting started

git clone https://github.com/mraza007/codewiki.git
cd codewiki
cargo install --path .

cd your-project
cw init
cw setup claude-code

Start a Claude Code session and the skill handles the rest. The project is MIT licensed and on GitHub.

Coding agents help with this. I can point Claude Code at an issue and let it work while I do something else. But that’s still one agent, one issue, one terminal. If I have 10 issues labeled “agent-ready,” I’m not babysitting 10 terminal tabs.

I wanted something that just watches for new issues and sends agents after them. Then OpenAI released their Symphony spec, an orchestrator pattern for their Codex agent. The architecture was solid: poll an issue tracker, dispatch agents into isolated workspaces, reconcile when issues close. But it was built around Codex and Linear, and I use Claude Code and GitHub Issues.

So I took the ideas I liked from Symphony and built my own. That’s Baton.

What it does

Baton is a Python daemon. You start it in your repo, it polls GitHub Issues matching your configured labels, creates an isolated git worktree per issue, and runs Claude Code CLI as a subprocess. When the agent finishes and opens a PR, Baton releases the claim and grabs the next issue.

One config file. One command. Go do something else.

WORKFLOW.md -> Orchestrator -> Worker (per issue)
                  |              |
                  |              +-- git worktree create
                  |              +-- hooks (before_run)
                  |              +-- claude -p "<prompt>"
                  |              +-- check issue state
                  |              +-- hooks (after_run)
                  |
                  +-- Poller (gh issue list)
                  +-- Dispatcher (concurrency control)
                  +-- Reconciler (stale run detection)

The name comes from relay races. You hand off the baton and the runner goes.

The config

Everything lives in WORKFLOW.md. YAML front matter for configuration, Jinja2 template below for the prompt. Baton reloads this file on every poll cycle, so you can change settings without restarting.

---
tracker:
  kind: github
  labels: ["agent"]
  exclude_labels: ["blocked"]

polling:
  interval_ms: 30000

agent:
  max_concurrent: 3
  max_turns: 5
  command: claude
  permission_mode: bypassPermissions

hooks:
  before_run: |
    git fetch origin main && git rebase origin/main
  timeout_ms: 60000
---

You are an autonomous software engineer working on issue #{{ issue.number " }}: {{ issue.title " }}.

{{ issue.body " }}

{% if attempt %}
This is continuation attempt {{ attempt " }}. Review what was done and continue.
{% endif %}

## Instructions

1. Understand the issue requirements
2. Write clean, well-tested code
3. Run existing tests to make sure nothing breaks
4. Commit your changes with a descriptive message
5. Push the branch and create a pull request linking to #{{ issue.number " }}

Labels filter which issues get picked up. max_concurrent controls parallel agents. max_turns is the retry limit per issue. Hooks run shell commands at different points. I use before_run to rebase on main so the agent starts from fresh code.

The prompt template gets issue.number, issue.title, issue.body, issue.labels, and attempt for retries. Standard Jinja2.

Why worktrees

Each issue gets its own worktree under .symphony/worktrees/, with a branch name slugified from the issue title: baton/fix-login-redirect-42.

I thought about Docker containers and temp directories but worktrees won out. They share the git object database so creating one is almost instant, unlike a full clone. They’re real checkouts, so linters and test runners and build scripts all work without any path hacking. And they’re isolated. If one agent trashes its branch, the others don’t care.

Why gh CLI instead of the GitHub API

Baton shells out to gh issue list and gh pr create instead of using PyGitHub or the REST API. Seems odd, but think about setup.

With the API, you need a personal access token. You need to configure it somewhere. You need to handle rate limits.

With gh, you authenticate once (gh auth login) and everything on your machine uses the same credentials. No token management in the orchestrator. The tradeoff is speed, but Baton polls every 30 seconds. The overhead of a subprocess call doesn’t matter at that pace.

The permission problem

This tripped me up. Claude Code has permission modes: default asks for everything, acceptEdits auto-approves file edits but prompts for shell commands, and bypassPermissions auto-approves everything.

I started with acceptEdits because it felt like the right balance. Let the agent write code freely, but make it ask before running commands. Problem: “ask” means a human clicking yes, and in an autonomous orchestrator there’s no human. The agent just blocks forever waiting for a prompt nobody will answer.

I wasted about 20 minutes watching it hang before I figured this out. For autonomous operation you need bypassPermissions, which maps to --dangerously-skip-permissions. The flag name is honest about the risk. I’m comfortable with it because the agents run in isolated worktrees on disposable branches, not in my main checkout.

Auto-releasing on PR creation

My first version had a dumb problem. The agent would finish its work, create a PR on turn 2 of 5, and Baton would keep scheduling continuation turns for the remaining 3. The slot was occupied but nobody was doing anything useful.

The fix: after each worker finishes, check if a PR exists for that issue’s branch. If yes, release the claim immediately and free up the slot. If not, schedule a short retry.

pr_exists = await self.tracker.check_pr_exists(issue.number)
if pr_exists:
    log.info(f"PR_READY #{issue.number} -- PR found, releasing claim")
    return "pr_created"
return "no_pr"

Small change, but it meant the orchestrator stopped wasting time on finished work.

Extensibility through skills and MCP servers

Baton itself is deliberately simple. It polls, dispatches, and manages worktrees. The interesting part is what you put in the prompt and what tools you give the agent.

Claude Code supports MCP servers, which means you can wire up external tools and the agent can use them during its run. Baton passes MCP server config through to each worker:

agent:
  mcp_servers:
    - name: playwright
      command: npx @playwright/mcp@latest

That means the agent has access to a headless browser while it works. It can open a page, click around, take screenshots, verify that the UI renders correctly. You don’t have to build that into Baton. You just declare which MCP servers you want and the agent figures out when to use them.

Same idea with CLI tools. If agent-browser is installed on the machine, you can tell the agent to use it in the prompt template. “Before creating a PR, open the app with agent-browser and verify the acceptance criteria.” The agent spins up a local server, opens the page, clicks buttons, fills inputs, takes snapshots. All from instructions in WORKFLOW.md, nothing hardcoded in the orchestrator.

Claude Code also has skills, which are reusable prompt fragments that teach the agent specific capabilities. If you have a code review skill or a testing skill installed, the agent can use them during its run. Baton’s config supports a skills list for this:

agent:
  skills:
    - code-reviewer
    - accessibility-checker

You can also override skills per issue by adding a ## Skills section to the issue body. If one issue needs Playwright but the others don’t, just add it to that issue.

The point is that Baton doesn’t need to know about browsers or test runners or linters. It just needs to dispatch agents with the right config. The prompt and the tools do the rest.

Putting it together: a todo app from scratch

To see all of this working end to end, I had Baton build a todo app. Fresh repo, no code. I created three GitHub issues labeled baton:

1. Create basic HTML structure
2. Add JavaScript for create/delete
3. Add localStorage persistence

The WORKFLOW.md prompt told the agent to use agent-browser for verification before opening PRs. I ran baton start and went to make coffee.

Baton picked up issue #1, created a worktree on baton/create-basic-todo-app-html-structure-1, and dispatched Claude Code. The agent wrote index.html, spun up a local server with npx serve, opened it with agent-browser, confirmed the layout rendered, then committed, pushed, and opened a PR. The PR description included what agent-browser found:

Opened http://localhost:3456 and confirmed the page renders correctly. Ran agent-browser snapshot -i confirming interactive elements: textbox and button.

I merged it. The issue auto-closed (the PR had Closes #1). Baton saw the issue was gone on the next poll, released the slot, and picked up issue #2. Same cycle. Then #3.

Three issues, three PRs, three merges. I didn’t write a line of the todo app. The agent-browser verification wasn’t built into Baton. It was just instructions in the prompt and a CLI tool on my machine.

Getting started

pip install -e .
cp WORKFLOW.md.example WORKFLOW.md
# Edit WORKFLOW.md: set your labels, tweak the prompt
baton start

You need Python 3.11+, Claude Code CLI (claude), GitHub CLI (gh) authenticated, and Git.

The code is at github.com/mraza007/baton. MIT licensed. About 10 Python modules, no external services, no databases. State lives in memory with JSON persistence for the status command.

What I want to add next

• A proper TUI instead of baton status reading a JSON file
• Issue dependency ordering so issue 3 waits for issue 2 if it needs to
• Cost tracking per issue, so I can see what automating the backlog actually costs in tokens
• More trackers besides GitHub Issues (Linear, Jira, GitLab)

If you’ve got a repo with a pile of issues sitting there, try pointing Baton at it. Start with one label and max_concurrent: 1. See what it does. The setup takes about five minutes and the worst case is you get a bad PR that you close. The code is MIT licensed, the whole thing is ten files, and there’s nothing weird in it. Fork it, break it, rip out the parts you don’t like.

If you try it, I want to hear what breaks.

I write a newsletter called Dev Console where I cover what’s actually happening in AI, minus the hype. New tools, real use cases, stuff I’m building. If this post was interesting, you’ll probably like it.

Harness engineering. It’s the practice of building everything around the LLM — the execution environment, tool definitions, safety boundaries, observability, and lifecycle management. The stuff that turns a chatbot into a production system.

If you work in DevOps, you’ve been doing this for years. You just called it something else.

Why Harnesses Matter More Than Models

Pick any AI agent demo. Strip out the model. What’s left?

A container or sandbox. A set of callable tools. A loop that reads output and decides what happens next. Logging. Timeouts. Cleanup.

That’s the harness. And it’s where agents succeed or fail. A great model in a bad harness hallucinates, loops forever, leaks secrets, or silently does nothing useful. A decent model in a good harness stays bounded, recovers from errors, and produces auditable results.

DevOps engineers already think this way. You don’t just pick a good application — you build the infrastructure that makes it reliable. Same thing here.

The Five Parts of a Harness

Here’s how I break down harness engineering into components. Each one maps directly to something you’ve built before.

1. Execution environment. Where does the agent run? A container, a VM, a temporary directory, a git worktree. You need isolation so the agent can’t corrupt shared state. You need reproducibility so runs are consistent. This is the same problem as CI job runners. Docker, Firecracker, nsjail — pick your isolation boundary.

2. Tool definitions. Tools are the agent’s API surface. Read a file. Run a command. Query a database. Call an endpoint. Each tool needs input validation, output formatting, error handling, and permission scoping. Think of it like designing an API — you wouldn’t expose raw database access through a REST endpoint. Don’t give an agent raw shell access either. The tool layer is your contract.

3. Control loop. Observe, decide, execute, verify. The loop is what makes an agent an agent instead of a one-shot prompt. Your job as a harness engineer is to decide: how many iterations? What’s the timeout per step? What happens when a tool call fails? When does the loop escalate to a human? This is the same logic you put in health check loops and deployment rollback controllers.

4. Guardrails. Cost caps. Token limits. Command allowlists. File path restrictions. Rate limiting on external calls. Without guardrails, an agent can burn through your API budget in minutes or write to paths it shouldn’t touch. Every guardrail is a policy decision — same as IAM policies, network rules, and resource quotas you already manage.

5. Observability. If you can’t see what the agent did, you can’t debug it, audit it, or trust it. Log every tool call, every LLM response, every decision point. Capture diffs, timing, token usage, and cost. This is no different from structured logging in any production system. The difference is that agent traces are longer and less predictable than HTTP request traces, so you need good tooling to navigate them.

Where DevOps Context Overlaps

Here’s where your existing skills plug in directly.

Infrastructure as code. Agent harnesses should be declarative and version-controlled. The tool definitions, policies, and environment specs should live in config files, not hardcoded in application logic. When you change a tool’s behavior, that change should be reviewable in a PR.

Pipeline orchestration. Multi-agent systems look a lot like multi-stage pipelines. One agent does research, passes context to a planning agent, which passes a plan to an implementation agent. You’re managing handoffs, shared artifacts, and failure propagation — the same coordination problem as CI/CD stages.

Incident response. When an agent goes wrong, you need the same muscle memory. Check the logs. Find the failing step. Understand the input that caused it. Roll back if needed. The debugging workflow is identical.

Security boundaries. Least privilege applies to agents just like it applies to services. What tools can this agent access? What files can it read? Can it make network calls? Can it spend money? Every agent needs a security boundary, and DevOps engineers already think in terms of boundaries.

Getting Started

If you want to start building harnesses, you don’t need a new framework. Start with what you have.

Take a simple task — say, analyzing a failed CI build. Write a script that collects the logs, sends them to an LLM with a prompt, parses the response, and posts a summary to Slack. That’s a harness. A minimal one, but it has all the components: environment setup, tool use (log collection, Slack posting), a control flow, and output handling.

Then add complexity. Let the LLM decide which logs to fetch. Add a retry loop. Add a cost cap. Add structured logging. Each addition is a harness engineering decision.

You don’t need to learn ML. You don’t need to fine-tune models. You need to build the infrastructure that makes models useful — and that’s the job you already do.

Harness engineering isn’t a new discipline. It’s DevOps applied to a new kind of workload. The sooner you see it that way, the faster you’ll build agents that actually work in production.

You start a new session and it re-discovers the same patterns. Repeats the same mistakes. Asks the same questions. It’s like working with a brilliant colleague who gets amnesia every night.

I got tired of this. So I built EchoVault — a local memory system that gives coding agents persistent memory across sessions. No cloud. No API keys. No cost. Just a SQLite database and some Markdown files on your machine.

The Problem Is Real

I use coding agents daily across multiple client projects. Claude Code, Cursor, Codex — I switch between them depending on the task. Every time I start a session, I’m repeating context that the agent should already know.

“We chose FastAPI over Flask because of async support.” “The deploy script needs –no-cache or the CSS breaks.” “Don’t touch the legacy auth module — it’s being replaced next sprint.”

I was copy-pasting this stuff into every session. That’s not how tools should work.

I tried existing solutions. Supermemory announced their MCP and I was tempted, but it saves everything in the cloud. I work with multiple companies as a consultant — I don’t want codebase decisions stored on someone else’s servers. Claude Mem was the first tool I tried, but it was eating too much memory in my sessions and became a bottleneck when running multiple agents at the same time.

So I built my own.

How EchoVault Works

EchoVault runs as an MCP server. When your agent starts a session, it has three tools available:

• memory_context — load prior decisions, bugs, and context for the current project
• memory_search — find specific memories by keyword or semantic similarity
• memory_save — persist a decision, bug fix, pattern, or learning

The agent calls these tools like it calls any other tool. No hooks. No shell scripts. No prompt injection. The MCP protocol handles everything.

Here’s what happens in practice:

Session start. The agent sees memory_context in its available tools. The tool description says “You MUST call this at session start.” The agent calls it and gets back a list of prior memories for the project. Now it knows what happened yesterday.

During work. You ask about authentication. The agent calls memory_search with “authentication” and gets back the decision to use JWT, the bug with token refresh, and the migration plan. It has context before writing a single line of code.

Session end. The agent just fixed a tricky race condition. The tool description says “You MUST call memory_save before ending any session where you made changes.” It saves the root cause, the fix, and what to watch for.

Next session, that knowledge is there. Every session builds on the last one.

The Architecture

I kept it simple. The whole system is four things:

~/.memory/
├── vault/                    # Obsidian-compatible Markdown
│   └── my-project/
│       └── 2026-02-01-session.md
├── index.db                  # SQLite: FTS5 + sqlite-vec
└── config.yaml               # Optional embedding config

Markdown vault. Every memory gets written to a session file — one file per day per project. These are valid Markdown with YAML frontmatter. You can point Obsidian at ~/.memory/vault/ and browse your agent’s memory visually. You can read them in any editor. They’re not locked in a proprietary format.

SQLite index. This is where search happens. FTS5 handles keyword search out of the box — no configuration needed. If you want semantic search (where “authentication” matches a memory titled “JWT token setup”), add an embedding provider. I use Ollama with nomic-embed-text locally. You can also use OpenAI or OpenRouter if you prefer cloud.

MCP server. The agent talks to EchoVault through the Model Context Protocol. Three tools, stdio transport, nothing fancy. The server starts when the agent needs it and stops when the session ends. Zero idle cost.

Secret redaction. Three layers. Explicit <redacted> tags for things you mark yourself. Pattern detection that catches API keys, passwords, and credentials automatically. And .memoryignore rules for custom patterns. Nothing sensitive hits disk.

Making Agents Actually Save

Here’s the thing about MCP tools — the agent can call them, but will it? Retrieval works well because agents tend to grab context at the start. Saving is the hard part. The agent finishes its work and moves on. It doesn’t naturally think “I should save what I learned.”

The trick is the tool descriptions. When you register an MCP tool, you include a description. Agents read these descriptions and treat them as instructions. So instead of:

"Save a memory for future sessions. Call this when you make decisions."

I wrote:

"Save a memory for future sessions. You MUST call this before ending
any session where you made changes, fixed bugs, made decisions, or
learned something. This is not optional — failing to save means the
next session starts from zero."

That “MUST” language makes a real difference. It’s not 100% reliable — nothing with LLMs is — but agents follow strong tool descriptions much more consistently than passive ones.

Cross-Agent Memory

One of the things I wanted was a single vault for all my agents. A memory saved by Claude Code should be searchable from Cursor or Codex. They’re all working on the same codebase. Why should they have separate memories?

EchoVault stores everything in one place. The MCP server is the same regardless of which agent connects to it. Setup is one command per agent:

memory setup claude-code   # writes ~/.claude.json
memory setup cursor        # writes .cursor/mcp.json
memory setup codex         # writes .codex/config.toml + AGENTS.md
memory setup opencode      # writes opencode.json

Each agent has its own config format and conventions. Claude Code uses JSON with mcpServers. Cursor uses the same schema but different file paths. Codex uses TOML with [mcp_servers]. OpenCode uses JSON with a mcp key and a different command format (command as an array instead of separate command + args).

I wrote shared helpers so each agent’s setup is just a thin wrapper around _install_mcp_servers() or _install_toml_mcp(). Adding a new agent takes maybe 20 lines of code.

What Gets Saved

Not everything should be a memory. Trivial changes don’t need to be persisted. Information that’s obvious from reading the code doesn’t need a memory. The goal is to capture what a future agent wouldn’t know from just looking at the codebase.

Good memories:

• Decisions. “Chose JWT over sessions because the API needs to be stateless.” A future agent reading the code sees JWT but doesn’t know why.
• Bugs. “The ORM lazy-loads relationships by default, causing N+1 queries in the user list endpoint. Fixed by adding .options(joinedload(...)). Root cause: SQLAlchemy default behavior.” A future agent won’t hit the same bug.
• Patterns. “All API endpoints follow the pattern: validate input, check permissions, execute, return response. Don’t add business logic in the route handler.” A future agent follows the existing patterns instead of inventing new ones.
• Context. “The legacy auth module is being replaced. Don’t modify it — changes go into the new auth service at src/auth/v2/.” A future agent doesn’t waste time on dead code.

Each memory has a title, a “what happened” summary, optional “why” and “impact” fields, tags, and a category. Search returns compact ~50-token summaries. Full details are fetched on demand so context windows don’t get bloated.

The Technical Bits

A few implementation details that might be useful if you’re building something similar.

FTS5 for keyword search. SQLite’s FTS5 extension is fast and works with zero configuration. No external service needed. It handles stemming, phrase matching, and ranking. For most use cases, this is all you need.

sqlite-vec for semantic search. When you want “authentication” to match “JWT token rotation”, you need vectors. I use sqlite-vec to store embeddings right in the same SQLite database. No vector database needed. Embedding providers are pluggable — Ollama for local, OpenAI or OpenRouter for cloud.

Hybrid search. The search pipeline runs FTS5 first (fast, precise), then semantic search (slower, fuzzy), and merges the results. This gives you the best of both worlds — exact keyword matches and semantic similarity.

TOML parsing with fallbacks. Codex writes some non-standard TOML — unquoted filesystem paths as table keys, dotted version strings as key names. Standard tomllib chokes on these. I added a fallback that appends the MCP section directly via string operations when parsing fails. It’s not pretty but it handles real-world config files.

Symlink handling. Some agents create symlinks in their skill directories. shutil.rmtree() crashes on symlinks. Small thing but it bit me in production.

Setting It Up

pip install git+https://github.com/mraza007/echovault.git
memory init
memory setup claude-code

That’s it. Three commands. The agent has memory now.

If you want semantic search, configure an embedding provider:

memory config init
# Edit ~/.memory/config.yaml to set your provider
memory reindex

For fully local operation with no external API calls, use Ollama:

embedding:
  provider: ollama
  model: nomic-embed-text

What I’ve Learned

Building this taught me a few things about agent tooling.

Tool descriptions are instructions. Agents read them and follow them. Strong, directive language in tool descriptions is more effective than passive documentation. “You MUST” works better than “You can.”

Local-first matters. Not because of ideology, but because of practical constraints. Consultants work with multiple clients. Sensitive decisions shouldn’t leave the machine. And when your internet goes out, local tools still work.

MCP is the right abstraction. Instead of writing agent-specific hooks, skills, and config formats, I write one MCP server and each agent connects to it. When a new agent comes along, I add a setup function for its config format. The memory logic doesn’t change.

Simple storage wins. Markdown files you can read in any editor. SQLite you can query with any tool. No custom binary formats. No daemon to keep running. The system is completely inspectable and debuggable.

The code is at github.com/mraza007/echovault. It’s MIT licensed. If you’re tired of your agents forgetting everything, give it a shot.

Now swap out the hardcoded logic for an LLM. That’s it. That’s an AI agent in simpler terms. The fancy demos want you to think it’s magic. Some brand new thing you need to learn from scratch. It’s not. When you take away the hype, an agent is just a controlled automation loop. The LLM handles the reasoning and everything else is infrastructure you’ve built a hundred times.

Here’s what matters, the agent itself isn’t the hard part but The harness is, the execution environment, tooling, guardrails, and observability. It’s all the important stuff that makes automation work in production.

DevOps engineers have been building harnesses forever. CI runners. Deployment pipelines. Infrastructure automation. The patterns are the same. The skills transfer directly.

So if you’re wondering whether AI agents are worth learning, here’s the short answer. You’re already halfway there.

What an Agent Actually Looks Like

Let’s forget the marketing hype around AI agents and understand from a DevOps engineer’s point of view, what an agent actually looks like. An AI agent has six parts.

1. An LLM: Now LLM is the most important part of an agent as this acts as a brain. It reads context and decides what to do next. It doesn’t touch anything directly.

2. A workspace: Think of it as a sandboxed environment. A cloned repo. A container. A temp directory. Same as any CI job.

3. A set of tools: These are the actions it can request. Read a file. Run a command. Call an API. Query logs. The agent doesn’t run these itself. It asks for them.

4. A control loop: This is the core pattern. Observe the current state. Decide an action. Execute it. Check the result. Keep going until you’re done.

5. Policies and limits: Timeouts. Permission boundaries. Rate limits. Cost caps. Without these, agents can spin forever or do things they shouldn’t.

6. A termination condition: The agent needs to know when to stop. Task complete. Error threshold hit. Human review needed. Something has to end the loop.

Now none of this is new as you’ve built systems with all these components. The only difference is the LLM sitting in the decision seat.

The Harness Does the Heavy Lifting

Everyone focuses on the LLM. They miss the important part. The harness is what makes an agent actually work.

The harness is everything around the model. It spins up the environment. Exposes tools. Executes commands on the agent’s behalf. Captures logs and diffs. Enforces limits. Decides when the loop should stop.

Sound familiar? It should. This is what CI runners do.

GitHub Actions. GitLab runners. Jenkins agents. They all follow the same pattern. Spin up an isolated environment. Run steps. Capture output. Handle success and failure. Clean up.

An agent harness does the exact same thing. The only twist is the steps aren’t hardcoded in YAML. They come from the LLM at runtime.

This is why DevOps engineers are perfect for this work. You already think about isolation, execution, logging, and cleanup. You already build systems that run untrusted code safely. Agent harnesses are the same problem with a new input source.

Tool Use Is the Safety Mechanism

Agents don’t touch systems directly. This matters. The LLM never runs a command itself. Never writes a file itself. It requests actions through tools.

The harness gets the request. Validates it. Executes it in a controlled way. Returns a structured result.

This is how you keep agents safe.

Say the agent wants to run a shell command. The harness can check it against an allowlist. Run it in a sandbox. Set a timeout. Capture stderr. The agent never gets raw shell access.

Same thing for file operations. The agent requests a file write. The harness checks the path. Validates the content. Writes the file and returns confirmation.

You control what tools exist. You control how they behave. You control what the agent can even ask for.

This is the same idea behind least privilege. The agent only gets access to what it needs. The harness enforces the boundary.

The Control Loop in Practice

The core of any agent is the control loop. It looks like this.

1. Observe: The agent reads the current state. Test output. Log files. Diffs. Error messages. Whatever context it needs.

2. Decide: The LLM looks at the state and picks an action. Run another test. Edit a file. Ask for more information. Give up.

3. Execute: The harness runs the requested action and returns the result.

4. Verify: The agent checks if the action worked. Did the test pass? Did the error go away? Is the task done?

5. Repeat: If the task isn’t complete, go back to observe.

This loop keeps running until a termination condition hits—success, failure, timeout, max iterations, or human intervention.

You’ve seen this before: build, test, fix, rebuild. CI pipelines do this, deployment rollbacks do this, and health check loops do this.

Agents just make the “decide” step dynamic instead of scripted, and here’s where they actually help in DevOps work.

CI failure analysis. When a test fails, the agent reads the logs, checks the diff, identifies the cause, and suggests a fix—maybe even applying it and rerunning the test.

Terraform drift detection. The agent compares actual state to declared state, flags the drift, and proposes a remediation plan while a human approves before anything changes.

Kubernetes manifest review. The agent checks YAML against best practices (missing resource limits, no liveness probes, exposed secrets) catching the stuff humans miss in review.

Cost anomaly investigation. When spending spikes, the agent queries cost explorer, correlates with recent deployments, and surfaces the likely cause, saving an hour of digging.

Incident log triage. Faced with pages of logs, the agent reads them, extracts the relevant lines, and summarizes what went wrong (not replacing the engineer, but getting them to the answer faster).

Notice the pattern: the agent assists and handles the tedious parts while the human stays in control of decisions that matter.

AI agents sound complicated with their new frameworks, new terminology, and new paradigms.

But look past the hype and you’ll see something familiar.

An agent is an automation loop where the LLM picks the next step, the harness executes it safely, tools provide controlled access to systems, and policies keep things bounded.

This is CI/CD architecture, infrastructure thinking, the stuff you already do.

When you read about agent frameworks or watch demos of coding assistants, you now have a lens to see the harness underneath, spot the control loop, and ask the right questions: what tools does it expose, what limits exist, and how does it handle failure?

You don’t need to become an ML engineer to understand agents—you just need to recognize the infrastructure patterns you’ve been using all along.

The LLM is the new part. Everything else is your domain.

After one week of systematic analysis, I cut it to $2,500/month — a 50% reduction, saving them $30,000 annually. Here’s exactly how I did it, with the scripts you can use.

The Discovery Phase: How I Found the Problems

Before touching anything, I needed to understand the infrastructure. Here’s my systematic approach:

Step 1: Pull Cost Data by Service

First, I analyzed their AWS Cost Explorer data to understand where money was going:

aws ce get-cost-and-usage \
  --time-period Start=2024-11-01,End=2024-11-30 \
  --granularity MONTHLY \
  --metrics "BlendedCost" \
  --group-by Type=DIMENSION,Key=SERVICE

This gave me the high-level breakdown. But I needed more detail.

Step 2: Build a Resource Inventory

I wrote a Python script to scan all resources across regions and identify optimization opportunities:

import boto3

def scan_ebs_volumes():
    """Find GP2 volumes that should be GP3 and unattached volumes."""
    ec2 = boto3.client('ec2')
    volumes = ec2.describe_volumes()['Volumes']

    gp2_volumes = []
    unattached = []

    for vol in volumes:
        if vol['VolumeType'] == 'gp2':
            gp2_volumes.append({
                'VolumeId': vol['VolumeId'],
                'Size': vol['Size'],
                'State': vol['State'],
                'MonthlyCost': vol['Size'] * 0.10,  # GP2 pricing
                'GP3Cost': vol['Size'] * 0.08,      # GP3 pricing
                'Savings': vol['Size'] * 0.02
            })

        if vol['State'] == 'available':  # Not attached
            unattached.append(vol)

    return gp2_volumes, unattached

I built similar functions for:

• RDS instances (storage type, utilization, Multi-AZ necessity)
• EC2 instances (generation, Reserved Instance coverage)
• Elastic IPs (attached vs idle)
• EBS snapshots (age, associated volumes)
• S3 buckets (storage class, lifecycle policies)

Step 3: Analyze CloudWatch Metrics for Utilization

This is critical. Before recommending any right-sizing, I needed data:

def get_instance_utilization(instance_id, days=30):
    """Get average CPU utilization over the past N days."""
    cloudwatch = boto3.client('cloudwatch')

    response = cloudwatch.get_metric_statistics(
        Namespace='AWS/EC2',
        MetricName='CPUUtilization',
        Dimensions=[{'Name': 'InstanceId', 'Value': instance_id}],
        StartTime=datetime.now() - timedelta(days=days),
        EndTime=datetime.now(),
        Period=86400,  # Daily
        Statistics=['Average', 'Maximum']
    )

    return response['Datapoints']

The results were eye-opening:

• Two t3.xlarge instances averaging 12% CPU
• RDS storage at 95% free space
• Multiple log groups with no retention policy (storing terabytes)

Step 4: Map Dependencies Before Cutting

Before deleting anything, I mapped what depended on what:

• Which services used which Elastic IPs?
• Which applications wrote to which log groups?
• Which backups were actually needed for compliance?

This prevented the classic mistake of breaking production while optimizing costs.

The Starting Point

After the discovery phase, here’s what I was working with:

The distribution told me a lot. EC2-related costs (compute + EBS + networking) made up over 38% of the bill. That’s where I started.

Phase 1: Quick Wins (Implemented Same Day)

Release Idle Elastic IPs — Saved $50/month

My inventory script flagged 5 Elastic IPs with no association:

aws ec2 describe-addresses --query 'Addresses[?AssociationId==null]'

Someone had provisioned them for test environments that were deleted months ago. Classic ghost infrastructure.

Time to fix: 5 minutes.

Migrate EBS GP2 to GP3 — Saved $125/month

The script found 6,000+ GB across multiple EBS volumes still on GP2. GP3 costs 20% less and provides better baseline performance (3,000 IOPS vs GP2’s variable IOPS based on size).

aws ec2 modify-volume --volume-id vol-xxx --volume-type gp3

No downtime. Just a CLI command per volume.

Time to fix: 30 minutes for all volumes.

Set CloudWatch Log Retention — Saved $100/month

My scan found 20+ log groups with no retention policy — storing logs forever:

aws logs describe-log-groups --query 'logGroups[?retentionInDays==null].logGroupName'

Set production to 90 days, staging to 30 days.

Time to fix: 20 minutes.

Phase 2: The Big Discoveries

AWS Backup Running 24x More Often Than Needed — Saved $400/month

This was the most surprising find. When I pulled the backup plan configuration:

aws backup get-backup-plan --backup-plan-id xxx

I saw: hourly backups. 24 backups per day. For every resource.

The backup storage had grown to $500/month — 10% of their total bill.

I reviewed their recovery requirements (they only needed daily backups with 14-day retention) and reconfigured:

{
  "ScheduleExpression": "cron(0 5 * * ? *)",
  "StartWindowMinutes": 60,
  "CompletionWindowMinutes": 120,
  "Lifecycle": {
    "DeleteAfterDays": 14
  }
}

Time to fix: 1 hour (including testing).

CloudWatch Metric Streams to Nowhere — Saved $400/month

My CloudWatch cost breakdown showed $400/month on “Metric Streams” — 100+ million metric updates going somewhere.

aws cloudwatch list-metric-streams

Found a stream configured to send data to a third-party monitoring tool. When I asked about it, nobody on the team knew it existed. The integration had been set up by a previous contractor and was never used.

This is a perfect example of ghost infrastructure that accumulates over time.

RDS Over-Provisioned by 95%

My RDS analysis showed all instances had massive storage allocated. The CloudWatch metrics told the real story:

aws cloudwatch get-metric-statistics \
  --namespace AWS/RDS \
  --metric-name FreeStorageSpace \
  --dimensions Name=DBInstanceIdentifier,Value=production-db \
  --start-time 2024-11-01T00:00:00Z \
  --end-time 2024-11-30T23:59:59Z \
  --period 86400 \
  --statistics Average

Result: 95% free space across all databases.

RDS storage can only be increased, not decreased. But I migrated all instances from GP2 to GP3 storage — same price, better performance.

For the next database refresh, I recommended right-sized storage instead of the default massive allocations.

Saved: $150/month

Phase 3: Infrastructure Improvements

NAT Gateway Consolidation — Saved $125/month

My VPC analysis showed NAT Gateways in every AZ across multiple regions costing $500/month combined. After reviewing their architecture and traffic patterns, they only needed half of them.

ECS Task Right-Sizing — Saved $250/month

The ECS service scan found:

• A staging service constantly failing health checks and restarting (consuming resources 24/7 while accomplishing nothing)
• Legacy services still running in production that nobody was using

These issues relate directly to the ECS architectural decisions that often waste weeks of engineering time. Plus, enabled Fargate Spot for fault-tolerant workloads (70% savings on those tasks).

S3 Lifecycle Policies — Saved $150/month

My S3 bucket analysis showed backup buckets had grown to 10+ TB with no lifecycle policy. Old backups were stored in Standard tier forever.

Added a policy to transition to Glacier after 90 days:

{
  "Rules": [
    {
      "ID": "archive-old-backups",
      "Status": "Enabled",
      "Transitions": [
        {
          "Days": 90,
          "StorageClass": "GLACIER"
        }
      ]
    }
  ]
}

Reserved Instances for Stable Workloads — Saved $500/month

My EC2 coverage analysis showed zero Reserved Instance coverage on instances running 24/7.

I helped them purchase Compute Savings Plans covering their steady-state workloads. Immediate 30-40% savings on compute.

EC2 Instance Right-Sizing — Saved $250/month

The utilization data was clear: multiple instances running at 10-15% CPU.

Downsized t3.xlarge instances to t3.large where utilization data supported it. Same workload, half the cost.

The Results

From $5,000/month to $2,500/month — exactly 50% reduction.

Over a year, that’s $30,000 back in their pocket.

The Methodology

Here’s the systematic approach I use for every cost optimization engagement:

1. Get the Data First

Before making any changes, I pull:

• AWS Cost Explorer data (by service, by tag, over time)
• CloudWatch metrics for utilization
• Resource inventory across all regions

2. Find the Ghosts

“Ghost infrastructure” costs more than you think:

• Unused Elastic IPs
• Detached EBS volumes
• Empty S3 buckets accumulating requests
• Log groups with infinite retention
• Metric Streams nobody monitors
• Test environments that outlived their purpose

3. Right-Size Ruthlessly

Check actual utilization before committing to Reserved Instances:

# EC2 CPU utilization over 30 days
aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 \
  --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-xxx \
  --start-time $(date -d "30 days ago" +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date +%Y-%m-%dT%H:%M:%S) \
  --period 3600 \
  --statistics Average

If a t3.xlarge averages 15% CPU, you’re paying for 85% idle capacity.

4. Modernize Storage

GP2 → GP3 is almost always worth it:

• 20% cheaper at baseline
• Better performance (3,000 IOPS baseline)
• Zero downtime migration

5. Review Backup Policies

Backups grow silently. Questions to ask:

• How often do you actually need backups?
• How long do you really need to keep them?
• Are you backing up dev/test environments at production frequency?

What This Looks Like Over Time

The best part? None of these changes affected performance or reliability. Most improved it.

Common Patterns I See

After doing this for multiple clients, patterns emerge:

1. Metric Streams nobody monitors — $100-400/month just disappearing
2. Hourly backups for daily restore needs — 24x the storage cost
3. GP2 volumes from years ago — never migrated to GP3
4. Multi-AZ staging databases — paying for HA nobody needs
5. NAT Gateways in every AZ — when one or two would suffice
6. Logs kept forever — “just in case”
7. No Reserved Instances — paying full on-demand for 24/7 workloads
8. Over-provisioned everything — “it might need it someday”

Need Help With Your AWS Bill?

I do AWS cost optimization as part of my DevOps consulting practice. If your AWS bill feels too high or you just want a second pair of eyes on your infrastructure, let’s talk.

Book a free 30-minute call — I’ll review your current setup and tell you where I see opportunities.

Have questions about any of these optimizations? Drop a comment below or reach out on Twitter/X.

The problem isn’t that ECS is hard. The problem is that teams treat infrastructure decisions like they’re permanent. They’re not.

Last year I worked with a startup that spent 5 weeks evaluating container orchestration options. Five weeks. They had a working app. They had paying customers waiting. But the engineering team was stuck in an endless loop of “what if we need to scale?” and “shouldn’t we future-proof this?”

They launched on Fargate. It took 3 days once they stopped debating.

Here are the 5 decisions that waste the most time and what I tell every client to pick.

Fargate vs EC2: Just Use Fargate

This one wastes more time than all the others combined.

I get it. EC2 looks cheaper on paper. You can run the numbers, build a spreadsheet, show that at 50 containers you’ll save $400/month with EC2. The finance person gets excited. Someone mentions spot instances. Now you’re three meetings deep into capacity planning for traffic you don’t have yet.

Here’s what actually happens with EC2: you spend a week figuring out instance types, another week on auto-scaling groups, then you hit some weird issue where your containers won’t place because the bin-packing algorithm can’t find space, and suddenly your “cheaper” option has eaten two sprints of engineering time.

Fargate just works. You tell it how much CPU and memory you need, and it runs your container. No instances to manage, no patching, no capacity planning.

“But it’s more expensive!”

Sure. Maybe 20-30% more at scale. But you’re not at scale. You’re trying to ship. And even if Fargate costs you an extra $200/month right now, that’s nothing compared to the $30k+ in engineering salaries you’re burning while debating this.

Python apps especially benefit from Fargate. Your Django app with Celery workers is memory-heavy and I/O bound. You’re not doing CPU-intensive work. Fargate lets you right-size memory without playing Tetris with EC2 instance types.

Pick Fargate. When you’re running 200 containers 24/7 and have real cost data, revisit. Until then, move on.

ECS Service Discovery: Use an Internal ALB

When your services need to talk to each other, AWS gives you three options: Cloud Map, internal ALB, or Service Connect. I’ve seen teams spend weeks evaluating all three, setting up proof-of-concepts, reading whitepapers.

Just use an internal ALB.

I know, it’s not great. It’s a load balancer. It’s been around forever. But that’s exactly why you should use it:

• It gives you a stable DNS name your services can call
• Health checks work out of the box
• You get access logs for debugging
• Every developer on your team already understands HTTP

Your FastAPI service calls http://api-internal.yourdomain.local/users and it just works. No service mesh. No Envoy sidecars. No DNS caching gotchas.

Cloud Map is fine, but I’ve debugged too many issues where services couldn’t find each other because of DNS TTL problems. Service Connect is powerful, but now you’re operating a service mesh. Do you really want to be debugging Envoy proxy configuration when your actual problem is a database query?

The internal ALB is boring. Boring is good. Boring means you’re debugging your application code instead of your infrastructure.

CI/CD for ECS: Use GitHub Actions

I’m gonna be honest here: if your code is on GitHub, use GitHub Actions. Don’t overthink this.

“But CodePipeline is AWS-native!”

Yes, and it requires you to set up a pipeline with Source, Build, and Deploy stages, configure IAM roles for each stage, create buildspec files, and wire everything together. It’s more YAML for the same result.

“But Jenkins gives us more control!”

It’s 2025. Please don’t set up a Jenkins server. You’ll spend more time maintaining Jenkins than deploying your app.

GitHub Actions has an official AWS action that handles ECS deployments:

- name: Deploy to ECS
  uses: aws-actions/amazon-ecs-deploy-task-definition@v1
  with:
    task-definition: task-definition.json
    service: my-service
    cluster: my-cluster
    wait-for-service-stability: true

That’s the whole thing. It registers your task definition, updates the service, and waits for the deployment to stabilize. AWS maintains it. It works.

Your deployment workflow lives in your repo, your team already knows GitHub Actions from running tests, and you’re not managing another piece of infrastructure. If you want to understand how these pipeline runners actually work internally, I wrote a deep dive on building a CI/CD pipeline runner from scratch in Python.

If you’re on GitLab, use GitLab CI. If you’re on Bitbucket, use Bitbucket Pipelines. The point is: use whatever’s already integrated with your code. Don’t add complexity.

ECS Secrets Management: Use SSM Parameter Store

Where do you store your database passwords and API keys?

Not in your task definition. I’ve seen that. Please don’t.

The two real options are SSM Parameter Store and Secrets Manager. Teams debate this endlessly because Secrets Manager has automatic rotation and sounds more “enterprise.”

Here’s the thing: SSM Parameter Store is free, integrates natively with ECS, and handles 99% of use cases.

{
  "secrets": [
    {
      "name": "DATABASE_URL",
      "valueFrom": "arn:aws:ssm:us-east-1:123456789:parameter/myapp/database_url"
    }
  ]
}

Your Python app reads os.environ['DATABASE_URL'] like it does locally. No SDK, no code changes.

Secrets Manager costs $0.40 per secret per month and is worth it if you need automatic rotation for RDS credentials. But you probably don’t need that on day one. Start with SSM, migrate specific secrets to Secrets Manager later if you need rotation.

And please, don’t set up HashiCorp Vault unless you have compliance requirements that specifically mandate it. You’re now operating a distributed system just to store passwords. That’s not simplifying your life.

ECS Logging and Monitoring: Use CloudWatch

Every team wants to evaluate Datadog, New Relic, Honeycomb, and then maybe self-host Prometheus and Grafana “for cost savings.”

Stop. Use CloudWatch.

Add this to your task definition:

{
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/ecs/my-service",
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "ecs"
    }
  }
}

Done. Your container logs go to CloudWatch. You can query them with Log Insights. Enable Container Insights and you get CPU/memory metrics. Set up a few alarms. You now have better observability than 80% of startups.

Datadog is genuinely good software. I like it. But it costs $15+ per host per month from day one, and you need to manage another vendor relationship. You can add it later when you actually need distributed tracing or APM.

Self-hosted observability is a trap. I’ve seen teams spend months building ELK stacks and Prometheus clusters. That’s infrastructure work that doesn’t ship features. Unless you have a dedicated platform team, don’t volunteer for this.

The Actual Pattern Here

Look at what I recommended:

• Fargate over EC2
• Internal ALB over Cloud Map or Service Connect
• GitHub Actions over CodePipeline or Jenkins
• SSM over Secrets Manager or Vault
• CloudWatch over Datadog or self-hosted

Every single choice optimizes for the same thing: less stuff to manage.

Yes, some of these cost slightly more money. Yes, some of them are less flexible. But they all share one property: they let you ship faster and debug easier.

And here’s what nobody puts in their architecture decision records: all of these choices are reversible.

• Fargate to EC2? Task definitions work on both.
• ALB to Service Connect? Just DNS changes.
• SSM to Secrets Manager? Same integration pattern.
• CloudWatch to Datadog? Add the agent, keep CloudWatch as backup.

The “wrong” choice costs you maybe a few hundred dollars a month in inefficiency. The debate about the “right” choice costs you weeks of engineering time.

What I’ve Actually Seen Happen

Teams that follow this advice ship in about a week:

• Day 1-2: Fargate cluster up, first service running
• Day 3: ALB routing traffic, services talking to each other
• Day 4: GitHub Actions deploying on push to main
• Day 5: Secrets in SSM, logs in CloudWatch, basic alarms set up

Week 2: Building features.

Teams that “do it right” are still having meetings about networking topology in week 6.

I’ve watched startups run out of runway while their infrastructure was still “almost ready.” I’ve seen senior engineers burn out on DevOps work instead of building the product that got them excited in the first place.

Your Python app on Fargate with CloudWatch logs isn’t going to fall over at 1,000 users. Probably not at 10,000. By the time scale is actually a problem, you’ll have the traffic data and revenue to solve it properly.

Ship first. Optimize later.

If you found this helpful, share it on X and tag me @muhammad_o7 - I’d love to hear about your ECS deployment experiences. You can also connect with me on LinkedIn.

Need Help? I’m available for AWS and DevOps consulting. If you’re stuck in ECS decision paralysis or need help getting to production faster, reach out via email or DM me on X/Twitter.