Reducing Feedback Latency with Local CI for Developers and AI Agents

We’ve spent a lot of time speeding up our cloud Continuous Integration (CI) runs. Improvements like better caching, faster machines, DAG reordering, and eliminating slow steps reduced the time to get feedback and merge new code to roughly 5–7 minutes. Yet even after all of that work, the feedback loop still felt too slow during local development. Cloud CI was still carried overhead from provisioning environments for each pull request, pauses between pipeline stages, reporting infrastructure, and the coordination costs of a distributed system, with each layer adding latency before we can act on the results.

Problem: Long Cloud CI Waits

After making any Pull Request, developers often faced a long enough wait for cloud CI that they would switch to another task while it ran. Even when CI completed, they typically wouldn't return immediately; instead, they would wait until the new task reached a natural stopping point before context-switching back to address any errors CI had surfaced.

In practice, this meant having a few different code changes lingering in every engineer’s mind even after the bulk of work was done. If LLMs were involved, their rate of iteration was quite slow as it waited longer between each change.

Solution: Local CI

Recent developer machines have gotten very powerful For example, the M4 Macbook Pro I’m writing this on has 14 cores and 48 Gigabytes of RAM, giving it comparable resources to our CI cluster used for a single commit. This ended up powerful enough to run the entire test suite for our smaller projects, and run almost all our steps of our main monolith quite quickly.

This reduced overhead across various areas:

• Setup: Bringing the CI loop locally removed setup overhead as a developer already had everything installed and running as part of the building out the feature, letting us skip to running the actual checks.
• Network: Running locally eliminated the network and coordination overhead inherent in cloud CI. Rather than pushing code to GitHub, waiting for GitHub to trigger our CI provider, and then waiting for results to propagate back through multiple services, developers received feedback directly on their machines.
• Worker allocation: In cloud CI, each step ran on a short-lived worker provisioned and coordinated by a shared cluster. This introduced several seconds of overhead between steps, along with occasional delays caused by resource contention from other builds competing for the same infrastructure.

Faster Feedback

Fast local feedback meant we no longer had to wait for cloud CI before feeling confident in a change. If local CI passed, cloud CI would typically pass as well. This faster feedback allowed for tighter iteration loops for developers and LLMs alike. In the past, only the relevant test would run after someone made a change, but now the entire CI was run after every change, catching issues faster.

Less Context Switching

Being able to just run CI and see results right away meant being able to fix issues immediately instead of waiting for a cloud run to finish. Getting immediate feedback led to faster iterations, both for developers and for workflows involving LLMs.

Despite spending considerable engineering effort optimizing our cloud CI, local runs still ended up being faster due to reduced overhead from competing resources, container initialization, compute allocation, and so on.

LLM Harness

At that point, we were already having LLMs work against our extensive test suite (written in RSpec and Jest) as a harness loop in some cases, but we would still encounter situations where code appeared to work fine, only to later fail cloud CI because an autogenerated type was missing, a linter rule missed, or a license check had failed.

Having a local CI allowed us to expand our harness to practically every step that could fail, allowing the LLM to iterate till an entire build passed. While this didn't guarantee we had the correct code, it did guarantee we had working and compliant code.

While it's also possible to enable this through MCPs and APIs with more complex permissions, allowing an agent to simply run a script, parse the output, and iterate on the results was much easier to integrate.

Making This Happen

Many of these steps were already in motion for their DevX or AgentX improvements, so their immediate positive impacts were also combined with the benefits that arose from a complete local CI.

Standardizing Local Environment with bin/setup

Ruby on Rails has an excellent command bin/setup that provides a consistent development environment in a single command for a new or pre-existing environment.

1. First, we converted our pre-existing onboarding scripts to the bin/setup pattern with idempotent steps, with a goal that developers simply just run $ setup daily that manages all the local development dependencies.
2. Then we progressively added our checkers that were only in the cloud as local dependencies so that everything was also available locally. As developers ran bin/setup occasionally, the local dev environment gained everything required to make local CI work.

The resulting bin/setup script used a mix of a list of project specific dependencies alongside building on top of Gemfile, package.json, Brewfile , and specific Docker images that ensured a consistent environment on every machine. While we created this to reduce developer annoyance for managing their local development set up and onboarding, this consistent environment per all machines established the pre-requisite for local CI.

Unifying Individual Steps to Work the same in cloud and local

Many of our Cloud CI steps were only written for our cloud build environment, but for this to succeed we needed every step to also work locally. As the local environment unified with our cloud environment, we started creating or rewriting local commands:

• For existing scripts: Cloud steps had to be rewritten in a more generic *nix format, with branches for system specific quirks if needed. In most cases these were quite quick to run through.
• For CI extensions: We had many steps that were run through a third party extension that ran the check. We rewrote these into simple scripts that worked on all systems.

The initial motivation was to help developers reproduce cloud CI failures locally and iterate more quickly. In the process, we made nearly every CI step available locally, which created a foundation for composing them in new ways.

Selecting the CI Runner

Once the local set up was consistent, we had the choice to figure out how to run our DSL. Our first instinct was to use a local runner for our cloud CI configuration, but first-party support never got too far beyond early stages and didn’t support our pipeline as-is, and third-party libraries for it never got too far either. So we decided to explore outside their ecosystem. We looked for:

• Minimal setup, ideally works out of the box
• Ability to run any command
• Summary Reporting - showing a summary of failures and successes for quick iteration

Your circumstances will definitely vary, so definitely check these out as part of your evaluation:

• Taskfile - The syntax and functionality looked quite familiar compared to our existing cloud CI configuration, and the cross platform support looked good, but we didn’t like introducing a new local dependency. Similarly, we also ruled out local Gitlab CI and GitHub Action runners.

Turning to what we had available already

• Shell scripts - These look great because they are familiar to developers, output each step with results (set -x), and quite easy to start with. For our CI logic, it started getting quite complicated and it didn’t have good out of the box summary reporting, though we initially did try building something simple. This is a great choice if your build is quite straightforward. For example, check out dhh/signoff.
• Rake - Ruby’s make-like task runner could work quite well since it already came bundled with rails and we were able to create a simple configuration that was more readable than the large shell script, but we still had to bolt on our own reporting summary code and aligning the dependencies. After we finished the project, we did find a nice community project called HellRok/LocalCI that extends Rake with some neat syntax to support this case.
• Rails bin/ci command - Recently, Rails 8.1 introduced an out of the box DSL that covered our needs for reporting and had a syntax that covered almost all our use cases with a neat bin/ci command. While it wasn’t as fleshed out as some of the other options we saw initially, it came out of the box while covering our cases.

CI.run do

  step "Style: Ruby", "bin/rubocop"

  step "Check: No TODOs", "if grep -r TODO app/; then exit 1; fi"

end

To run, it was simply:

$ ci

Rolling out the bin/ci DSL

After finalizing our choice of CI runner, we had several projects to migrate. We started with one of our recent Rails Projects that had just started with a simple CI check pipeline. Since it was up to date for Rails 8.1, we wrote our configuration in the recent DSL.

We got good initial feedback and fixed a few small bugs, and once that looked stable, we started to explore rolling it out to other projects. Many older projects were not at the latest Rails 8.1 yet, so we internally included the ActiveSupport::ContinuousIntegration file from Rails so that we could access it without upgrading the version. We did one project at a time, working up from the smallest to the biggest one. The biggest one, however, needed some extra support due to its size.

Extending CI DSL to support filter running

When we got to migrating our main monolith, we had to adapt around the platform CI run taking multiple hours due to the large number of tests and checks. Although local CI eliminated much of the overhead present in our cloud system and ran significantly faster, it still wasn't fast enough. We needed a way to be able to run one subset at a time.

A welcome bonus of including ContinuousIntegration was getting room to extend the DSL ourselves. For our purposes, we added a simple syntax for groups and a --group/-g command line option to run specific groups.

CI.run do
  group "backend" do
    step "Style: Ruby", "bin/rubocop"
  end

  group "frontend" do
    step "Biome: TypeScript", "bin/biome"
  end
end

Generally a developer/llm could figure out what subset to run, and the full distributed cloud CI run would run everything comprehensively. It was still quite an upgrade since a developer could run the parts of the CI relevant to their change to iterate quickly, so the odds of a cloud build failure still got to almost zero.

This was quite generally useful, so we took an attempt to upstream this.

Adopting bin/ci in the Cloud

Initially, we left the YAML files that defined our cloud CI configuration untouched, making local CI an optional, additive workflow. However, this meant maintaining two sources of truth for CI logic, which created the risk of developers updating one configuration but not the other, causing them to drift over time. Unification also gave developers and LLM agents more confidence that passing local CI will pass cloud CI.

Project by project, we replaced cloud CI steps with just simply calling bin/ci and passing the result through the cloud runner. While we weren't able to unify all the logic, we got almost all of it which helped reduce the potential for drift considerably.

Auto fixing upfront with bin/autofix

Most of our CI failures came from a dozen different linters, code generators, and truth table consistency checks that had fallen out of sync. While not every failure could be fixed automatically, the vast majority could. We consolidated all safe automatic fixes into a single command that applied every available fix, reported what had changed, and highlighted any remaining issues that still required manual intervention.

We had an earlier version of this based on bash scripts, but this was a good excuse to use the Rails CI DSL since it gave nice summaries and reporting that could be parsed easily. A developer or agent would quickly be able to fix the few that were not possible to fix.

This reduced the amount of iteration on local CI using a very similar set up, and it became common to simply run:

$ autofix && ci

Caveats

1. CI Size: Our CI pipeline is relatively easy to carve up, which allows us to run almost every step locally. That may not be true for every organization, especially larger engineering teams with more complex build systems. Even so, there is still significant value in running portions of a broader CI pipeline locally and allowing LLMs to iterate against those results.
2. Environment Parity: A local development environment is rarely identical to server environments, though I generally recommend unifying them as much as possible. While some teams use local CI runs as a replacement for cloud CI, we still run our cloud pipeline in a server-like environment as an additional safety layer. Local execution can also introduce security and compliance considerations around sign-off and verification.
3. Coverage: Not every CI step can be moved to a local environment, so a local CI run may never be fully comprehensive. However, it can still cover a substantial portion of a cloud CI pipeline and dramatically shorten feedback loops.
4. Runner Capabilities: The Rails CI DSL is intentionally lightweight. Rails 8.2 will add basic parallelism, but dedicated CI runners offer a much broader set of features. If your needs are more complex, there is a large ecosystem of local CI runners worth exploring. In our case, we were already using Rails and were excited to build on the new capabilities it introduced, and our big steps came with their own parallelism.

Takeaways

• Many of the improvements we made for agents ended up improving the developer experience as well, and vice versa. As a result, investments in developer experience became a rising tide that lifted both humans and agents.
• Feedback-loop latency matters when thinking about CI runtime. Even a highly optimized cloud CI system can feel slow if it interrupts a developer's train of thought, while immediate local feedback can have an outsized impact on productivity.

Cloud CI is still an important safety net for us, but we found that the most useful feedback is the feedback you get while you're still actively working on a change. By making our local environments more consistent, unifying how checks run locally and in the cloud, and making those checks easy for both developers and agents to execute, we were able to tighten that feedback loop considerably. As developer machines continue to get more powerful and agents become a more common part of the development workflow, we think local CI will only become more valuable. We spent years making our cloud CI faster, and that work was worthwhile. In the end, though, the biggest improvement came from fully leveraging the machines where developers and agents were already iterating.