The Hard Problem Isn't Automation. It's Classification.
James Coombs engineered an AI‑assisted bug pipeline that classifies tickets, dispatches them to specialized agents, and verifies fixes across backend, API, and frontend layers. By introducing a tiered system and a cross‑integration rule, the pipeline reduces manual effort and surfaces systemic issu…
James Coombs, a design engineer, recently unveiled an AI‑assisted bug pipeline that can ship full‑stack fixes across backend, API, and frontend layers in production. The system was tested on an “error handling improvement” ticket that required changes to a C# backend, a GraphQL schema, and a React frontend. What seemed like a simple 5‑hour task turned into 17 iterations of code changes, revealing the hidden complexity that often stymies automation.
From Ticket to Fix: The Four‑Stage Pipeline
The pipeline follows a clear sequence: Select, Classify, Dispatch, and Monitor. In the Select stage, the system pulls open tickets from the project tracker, filters them, and ranks them by estimated complexity. This step is straightforward and mirrors what any manual triage process would do.
Next comes Classify. Here the AI reads the ticket description, examines the relevant code, and assigns a tier. The tiers map directly to the level of human involvement and expected turnaround time:
- Tier A – A single‑layer fix with a clear scope. No human review is needed until the pull request merges, typically taking about 70 minutes of wall clock time and only five minutes of human oversight.
- Tier B – A fix that crosses integration boundaries. Human review of the pull request is required, usually taking four to five hours.
- Tier C – Investigation‑only tickets. A human implements the plan, saving 30 to 60 minutes of analysis time.
The Dispatch stage routes the ticket to the appropriate AI agent, pre‑loading it with domain‑specific context such as build commands, CI failure patterns, and cross‑layer rebuild sequences. The agent then attempts to generate a solution. Finally, the Monitor stage re‑runs critical gates independently after the agent finishes, ensuring that the agent’s “done” status matches the developer’s expectations.
Why Classification Matters More Than Automation
Coombs notes that simply throwing every ticket at an AI agent fails because the real challenge is determining which tickets are suitable for automation, how much supervision they require, and when to halt the agent before it wastes hours on a task that ultimately needs a human touch. The pipeline’s tiered approach addresses this by filtering out tickets that would otherwise lead to wasted effort.
A key rule in the system is that any ticket touching three or more integration layers—such as a backend ORM, an API schema, and a frontend component—is automatically reclassified to Tier B, regardless of the ticket’s stated complexity. This rule was born from a single ticket that, while titled “error handling improvement,” required changes across all three layers and took 17 iterations to resolve.
Results and Unexpected Discoveries
Across eight runs, the pipeline handled a range of full‑stack issues, including backend services, API resolvers, frontend components, and a security finding involving a missing authorization check. The CI failure pattern library, initially empty, grew to seven recurring patterns by run five, allowing the agent to apply known fixes on the first encounter instead of iterating.
Perhaps the most valuable outcome was the pipeline’s ability to surface systemic issues. During a Tier B run, the agent not only fixed the reported bug but also uncovered a widespread error‑handling pattern where API endpoints swallowed inner exceptions and returned raw error details to clients. In a later run, a UI alignment fix revealed two additional systemic problems: undersized form controls from a design system migration regression and redundant wrapper components that should have been caught by an ESLint rule.
These cascading discoveries—fix one bug, surface a pattern—are the highest‑value output of the pipeline, far outweighing the value of any single fix. By treating systemic discovery as a first‑class output, the pipeline transforms routine bug fixes into opportunities for proactive customer quality improvement.
What Could Be Done Differently
Coombs identifies several areas for improvement. First, framing the pipeline as a customer‑experience tool from the outset would shift its perceived value and resource allocation. Second, the CI failure pattern library should feed back into shared documentation so that human engineers can benefit from the knowledge the agent accumulates. Finally, the pipeline should explicitly output insights about what a ticket reveals about the codebase, not just the fix itself.
In practice, the first step to building a similar system is to start with classification. Sort the backlog into tiers, apply the three‑integration‑boundary rule, and then build the dispatch‑and‑verify loop for the simplest tickets. As the pattern library grows, the agent’s efficiency will compound, and the system will become increasingly capable of handling more complex, cross‑layer issues.
Why it matters
By turning tedious, multi‑layer bug fixes into automated, verified PRs, teams can reduce backlog, uncover systemic problems, and deliver higher‑quality code faster.
Key points
- AI pipeline uses a tiered classification to match ticket complexity with automation level.
- A rule that reclassifies tickets touching three integration layers prevents wasted effort.
- The system’s monitor re‑runs full CI gates to verify agent work, bridging the compliance gap.
- Systemic issues are surfaced as a high‑value output, turning fixes into proactive quality improvements.
- Starting with classification and a shared pattern library accelerates future automation.
- Framing the pipeline as a customer‑experience tool can change resource allocation and impact.
Frequently asked questions
What is the purpose of the three‑layer rule?
It ensures tickets that span backend, API, and frontend layers are treated as Tier B, requiring human review, preventing the AI from wasting time on complex cross‑layer fixes.
How does the monitor stage work?
After the AI agent completes a fix, the monitor re‑runs all critical CI gates—type checks, builds, linting—to confirm that the agent’s solution passes the full test suite.
