I Gave GPT‑5.6 Sol a 2,500‑Line Project and Asked It to Refactor Everything. 261 Files Later…

A practical report on using GPT‑5.6 Sol to rewrite a production MCP project. Not a benchmark. Not a review. Just what happened when I had 2,500 lines of TypeScript and one capable model.

I Didn't Expect It to Work This Well

I maintain a small open-source project called Agent Search MCP — a search router for AI agents. Twelve engine adapters, zero mandatory API keys, MCP protocol. Useful, but the code had grown organically over six months. Ad hoc error handling. Mixed responsibilities in the entry file. Every new engine meant copying and pasting the same HTTP boilerplate.

I'd been meaning to clean it up. But a full rewrite of a working project is hard to justify — you break things that work, you spend weeks, and the users don't care about your abstractions.

Then I got access to GPT‑5.6 Sol.

I didn't ask it to add features. I asked it to refactor the whole thing: unify the engine interface, extract shared infrastructure, fix the types, add tests for everything, and don't break the existing tools. Sol had the full codebase in its context — source, tests, docs, the whole thing.

What came back was the biggest single pull request I've ever reviewed: 261 files changed, 29,186 lines added, 14,289 deleted.

Here's what I learned.

What Sol Did That Genuinely Impressed Me

1. It Saw the Architecture, Not Just the Code

The old src/index.ts was 120 lines that did everything — create the server, register tools, set up HTTP, parse CLI flags. Every project has this file. Every developer knows it should be split. But no one does it because "it works."

Sol extracted it into five focused files:

I'd had this mental model for months. Sol wrote it in one pass.

2. It Genuinely Understands TypeScript Types

Before, the search provider type was a hand-maintained union:

type SearchProvider = 'duckduckgo' | 'sogou' | 'brave' | ...

Works fine until you add an engine. You'd update the type, then find five downstream type guards that also need updating.

Sol replaced this with a const array and derived type:

const SEARCH_PROVIDERS = ['duckduckgo', 'sogou', …, 'serper'] as const;
type SearchProvider = typeof SEARCH_PROVIDERS[number];

One source of truth. Every engine adapter, every tool registration, every fallback chain reads from it now. Adding Wiby, Bocha, Serper, and Tencent WSA was just adding four strings to one array.

3. It Coded Its Own Error Taxonomy

The old code threw generic Error('timeout') or Error('HTTP 429') strings. The new code has a structured error class with 10 failure types:

class EngineAdapterError extends Error {
  readonly failureType: 'validation_error' | 'parse_error' | 'timeout' 
    | 'upstream_4xx' | 'upstream_5xx' | 'rate_limited' 
    | 'bot_challenge' | 'permission_denied' | 'budget_exhausted' | 'unknown';
  readonly retryable: boolean;
  readonly cooldownMs?: number;
  readonly suggestion: string;
}

Every adapter now returns structured errors. The orchestrator knows which failures are retryable, which engines should be cooled down (60 minutes for Sogou's CAPTCHA page), and what suggestion to return to the AI agent. This wasn't in my prompt — Sol inferred it from the usage patterns.

4. It Generated Tests for Everything

Test files went from 44 to 73 — a 66% increase. Not just unit tests for the new infrastructure, but individual test files for every engine:

Before: tests/engines.test.ts  (monolithic, tested 2-3 engines)
After:  tests/engines/  (21 files, one per engine, all independent)

The benchmark framework was completely new — 9 files covering pooled comparison, quality metrics, runner qualification, and relevance calibration. Sol structured it as a capture/replay pipeline so results stay reproducible.

5. It Killed Code Without Sentiment

Fifty-four files deleted. Dead plans. Unused Python scripts. Outdated docs. A deprecated news tool that never worked reliably. A dedupByProvider() function that was declared but never called.

Most models are reluctant to delete. Sol wasn't. It identified dead code, deprecated it clean, and removed the file. That alone reduced cognitive load on the project significantly.

Where Sol Fell Short

It wasn't magic. A few things needed my intervention:

Verdict: Sol handles code structure brilliantly. It struggles with runtime behavior that requires real network testing and protocol-level safety.

The Numbers Tell the Story

Not every metric moved in the "right" direction. More source files isn't automatically better. But the structural quality — type safety, test coverage, error handling — genuinely improved.

So What's This Mean for AI-Assisted Coding?

One data point, not a conclusion.

What Sol does well:

• Cross-file consistency — 261 files without a single type break. That's genuinely hard for a human to do in one pass.
• Architecture extraction — identifying the right abstraction boundaries from a working codebase.
• Test generation at scale — not just covering the happy path, but building a whole benchmark framework.
• Code removal — harder for humans than most models, surprisingly.

What still needs a human:

• Runtime testing with real network conditions.
• Protocol-level safety (stdio, HTTP boundaries).
• Product decisions disguised as technical choices ("should we keep the news tool?").

Agent Search MCP v3.2 ships with these improvements today. Try it:

$ pnpm dlx agent-search-mcp

No API key. Eight free search engines. Provider routing mode so you control costs. And a type system that won't surprise you.

I'm lennney, a PM-minded AI engineer. I build tools at the intersection of AI agents and practical engineering. Read more at take-a-deep-breath0.com. The project is on GitHub and npm.