Technical field report · run date 2026-09-29

Silicon Exchange case study

Can a medium local model build useful software if the workflow around it is disciplined enough?

Most AI coding demos focus on the model. This one focuses on the system around the model.

The user supplied one app-level prompt; CodeLead ran the delivery work. Planning, the build, the checks, the repairs and the evidence capture all happened unattended after that. The model wrote the application code, inside a workspace CodeLead bounded and checked.

The question was never whether a local model could behave like a frontier model. It cannot. The question was whether a medium-sized local model could complete meaningful software work when CodeLead gave it the right operating environment: a plan checked against the request, a bounded build, verification independent of the model's own tests, scoped fixes, and evidence records.

That distinction matters. A powerful model can still fail if it is given too much freedom, too little context, or no validation discipline. A smaller one can produce surprisingly strong results when the work is planned, bounded, checked, and repaired.

The claim is not that local models replace frontier models for every task. It is that many engineering tasks do not need to start with the most expensive model available, and that with the right orchestration a medium-sized model can carry more of the work than most teams expect. That changes the economics of AI coding. It also changes the control model: instead of giving an autonomous cloud agent broad authority over a codebase, the work stays local-first, human-governed, and evidence-backed.

• 1h 46mlaunch to exit, unattended after the prompt; 1h 42m of it building
• 17 / 17increments verified
• 39 / 39model-written tests passing
• 30 / 30business-rule oracle cases passed, checked independently of the model's tests

The receipts are public. An application exactly as the pipeline produced it from this prompt, with its full git history: github.com/CodeLead-ai/silicon-exchange-app. The exact prompt, the run logs, the control arms, the other models, and an FAQ: github.com/CodeLead-ai/silicon-exchange-case-study. The application repository holds a run of 2026-09-14 with the same prompt and model, served by LM Studio. The artifacts of the run on this page (run journal, console and serving logs, oracle results) are being prepared for publication and will be added to the case-study repository soon; until then a run note there lists every figure with the artifact it was measured from. The CodeLead driver itself is part of the upcoming beta, so for now reproducing the result means running your own agent against the published prompt and comparing it with the receipts. If you do, on any hardware with any model, the repository has a template for a reproduction report, failures included.

What was tested

“Silicon Exchange” is a GPU rental marketplace from a public benchmark prompt featured in Alex Ziskind's video “I Gave Local AI and the Cloud the Exact Same Job”: five routes; five business-logic rule sets that must be pure, tested functions (half-open overlap detection, 15-minute round-up pricing with an excess-hours discount, hold expiry, maintenance blocking, combined filter and sort); URL-persisted filters; localStorage reservations; a dark “trading terminal” design language; and a quality bar of npm install · test · build · dev all green.

We ran the same functional requirements with one declared change: the stack was moved from Next.js to Vite, React and TypeScript, which is the stack the pipeline currently supports. The adapted prompt is published verbatim in the case-study repository.

What “unattended” means here

The user gave CodeLead that prompt, with the one declared stack change, and nothing else. From there CodeLead ran the delivery loop on its own: it planned the work and checked the plan against the request, then had the model build the planned increments one at a time, each one built, tested, probed in a browser and checkpointed before the next began. When an increment failed a gate, the failure went back to the model inside the same run, and the increment was checked again. After the run, every business rule in the request was checked against the built application independently of the model's own tests.

No person split the work, wrote application code, patched a failure, or decided when the application was done.

That is what the run is about. The claim is not simply that a local model generated code. It is that a governed workflow made a medium local model useful enough to deliver a complete, verified application from one app-level prompt.

Running hands-free was a property of this benchmark, not a recommendation for production work. In a modernization engagement people approve the plan and every risky change; see what people approved.

Why the test matters

The prompt was already used in a public comparison between a very large local model and a frontier cloud agent, which makes it a fair yardstick rather than a task chosen to flatter us.

Times are end to end as reported. The first two rows are as stated in the video; ours is measured from the run's own console log, first line to last.

The model

Qwen3.8 27B, the stock published weights with no fine-tuning, quantized to 8 bits and served by TensorFold with speculative decoding (a DFlash2 drafter) on a single MacBook Pro with an M5 Max chip and 64 GB of unified memory. It decoded at about 59 tokens per second (median 55, range 45 to 100 across the run's 52 requests). The model is roughly a hundred times smaller than the 2.8-trillion-parameter model in the reference comparison, which generated at 14.7 tokens per second across four machines.

What CodeLead controlled

The user's prompt asked for a whole application, but CodeLead did not hand that request to the model as one open-ended task and wait. It planned the work, bounded the build, judged the result at every boundary, and decided what was done. The model generated the code; CodeLead controlled the plan, the workspace, the checks, the fix and the evidence around it.

1. A plan in pieces

   One planning call decomposed the request into 17 dependency-ordered increments, with the pure business rules and their tests before the pages that use them. Planning took 1m 46s.

2. A check on the plan

   Before any building started, a coverage review and a plan critic checked the plan against the request so that stated requirements could not be silently dropped. Six were added. 2m 28s.

3. A governed build

   The model built one increment at a time inside a bounded, monitored workspace. CodeLead judged the result at every boundary, and the model never edited files without the harness seeing it.

4. A gate on every increment

   Each increment had to build, pass its tests and pass an acceptance probe in a real browser before it was checkpointed. Of 18 probe criteria, 11 were readable by a browser; 6 passed and 1 failed and was repaired.

5. Repair inside the run

   A failed gate sent the failure back to the model: 22 attempts for 17 increments, five retries, none of them by a person. Nothing was repaired after the run.

6. An independent check, and a record of everything

   After the run, the request's business rules were checked against the built application by an oracle written from the specification, not by the model: 30 of 30. Every increment was committed with its evidence.

Everything that makes the result Silicon Exchange was written by the model: the data layer, all five business-rule implementations, every page and component, and all 39 unit tests, 2,686 lines across 39 source files. The pipeline planned the work, bounded the build, ran the checks, sent the failures back, and recorded the evidence. It wrote no application code, and no person wrote or edited any.

The ledger below is the record of that loop rather than a hand-polished demo: each stage was checked and recorded before the next began, and the retries are in it.

What worked

• All 17 increments passed their checks and were committed. The run ended because the plan was complete, not because it was stopped.
• The model wrote 39 unit tests across five test files, all passing, and the request's business rules were then checked independently of those tests: 30 of 30 oracle cases.
• Four increments needed more than one attempt (three took two, one took three): two build failures, one test failure, one probe failure and one more build failure, every one repaired inside the run. Nothing was repaired after it.
• 48 model calls, 97.5 minutes of model time inside the 106 of the run, 758,514 prompt tokens and 257,486 completion tokens. Model cost: $0.00, because it ran locally.

The result

Screenshots as published in the case-study repository, from the run of 2026-09-14 with the same prompt and model; the run on this page was not captured. Reviewed by hand, and discussed under polish below.

Not tuned to this prompt

A fair worry about any benchmark result is that the tool was shaped around the benchmark. So, the same morning, we wrote two requests in unrelated domains, about 250 words and five business rules each, published them verbatim, and ran them through the same pipeline with nothing changed.

Two apps the pipeline had never seen, 15 of 15 increments verified, every business rule checked against its request, both reviewed by hand as good-looking. One thing both lacked was a way to add a new task or expense, and neither request asked for one. The pipeline built what was asked.

Why not just run the model raw?

The obvious question about any orchestration layer is how much of the result is the model. So we ran the same model, same prompt, same laptop and same inference settings with CodeLead removed, twice, with two different tools: opencode, the tool used in the video, and a plain agent loop of our own whose prompt is published line by line. “Raw” means the stock model, the public one-command Vite scaffold, and a loop that reads files, writes files and runs commands until the model says it is done. No plan, no gates, no verification beyond whatever the model chooses to run.

The plain loop produced a complete marketplace in 75 minutes: its six routes, 64 passing tests, build green, genuinely good-looking. And the pricing rule wrong: a booking a few minutes past 24 hours is discounted as if a whole extra hour had been used. Its own test asserts the wrong number. Every build, test and render check passed, and the run declared itself finished.

opencode, at the reasoning effort we use for planning, produced nothing: the model thought past its output budget without a single tool call and exited after 45 minutes with zero files. At low effort it wrote the business rules and 40 passing tests, no pages, and ended at 107 minutes on a client timeout. The loop shape that takes a 2.8-trillion-parameter model four hours does not finish on a 27B.

So why not raw? Because “finished” is the model's word for it. When code and tests come from the same misunderstanding, tests passing proves nothing, and nothing in the loop can tell you. In CodeLead, finished means checked: every business rule in the request is verified independently of the model's own tests. In the run above, that check passed 30 of 30 cases, and the failures the model did make on the way, in builds, a test and a browser probe, were caught at the gate and repaired before the increment could land, with nobody reading the code. That difference, not speed and not looks, is the case study. Settings and outcomes for both arms are in the control-arms document.

Other models, same pipeline

We ran ten models through the same pipeline, six on the laptop and four remote. Every increment that shipped was machine-verified, every one that did not is recorded as not built, and every finished app was also reviewed by hand.

Size, cost and result did not line up. The two remote models that completed the whole product were not the two that produced the best apps: GLM 5.3 verified everything and was the worst to use; DeepSeek and Kimi K2.7, at about a dollar a run, produced the best-looking apps of the series alongside the 27B on the laptop. Among local models, size alone predicted little: a 28B dense model got closer than a 35B mixture-of-experts with 3B active, and a 24B non-reasoning model produced a blank page. What separated the runs was whether the model could plan the whole request, hold a bounded task, and produce integer arithmetic, and the pipeline caught what each one missed. Remote rows ran on cloud hosts at three to five times the laptop's token rate, so their wall clocks are not comparable to the local ones. Per-model detail, the hand reviews and screenshots are in the other-models document.

Limits, tradeoffs, and what they mean

This was not a speed benchmark, and it was not a claim that medium local models replace frontier cloud agents for every software task. It was a test of a narrower idea:

Can a medium local model complete meaningful engineering work when CodeLead controls the workflow around it: the plan, the build, verification, repair, and evidence?

In this run the answer was yes, with real tradeoffs.

Speed: the cloud agent was faster, and that is not the whole story

The frontier cloud agent completed the challenge in fifteen minutes. This run took one hour and forty-six minutes from launch to exit. If elapsed time is the only goal, the cloud agent wins, and it is not close.

The experiment was not designed to show that local execution is faster. It was designed to test whether useful, verified software work can be done locally, under tighter control, without handing broad authority to a cloud agent.

The local comparison is the more interesting one. In the reference video, Kimi K3 took about four hours with a 2.8-trillion-parameter model served across four Mac Studios with 2 TB of unified memory. This run finished in well under half of that time, on one laptop, with a model roughly a hundred times smaller.

That does not mean Qwen is better than Kimi K3. One run on each side is not a benchmark, and this compares two whole setups rather than two models. Our reading of the video, not a measurement of their machines: an open-ended agent loop sends the whole growing conversation back to the model before every step, and a reasoning model thinks at length before each one, which is what an unstructured loop costs on a very large model. CodeLead spends the model differently, on one bounded job at a time, and verifies in the browser and the test runner instead of asking the model to think harder. It suggests that the workflow around the model matters.

For some teams, speed is what matters most. For others, keeping source code in-house, predictable cost, auditability, and human approval matter as much. CodeLead is built for the second group.

Polish: the frontier agent had the strongest visuals

In the reference comparison, the cloud agent's result was the most polished. That was expected: larger frontier models are still stronger at taste, visual judgment, and open-ended design.

Ours was reviewed by hand, and interface quality is a human judgment, not a metric. The other-models run made that plain: the model with the cleanest scorecard produced the least usable interface, and two of the best-looking apps came from models that left slices unverified. The screenshots and the video let you make your own judgment.

Visual polish was not what the run was testing, and a person can always give more specific design direction in a follow-up. For modernization work the distinction matters. The hard part is rarely a beautiful first draft. It is preserving behavior, controlling risk, and proving that each change is safe.

Repair was automated, not manual cleanup

The build did not land clean. Of the 17 increments, four needed more than one attempt: two failed their build, one failed its tests, one failed the browser probe, and one failed its build again on the second try. Twenty-two attempts for seventeen increments.

No one stepped in to patch any of them. Each failure stopped its own increment at the gate, went back to the model with the failure attached, and was checked again before the increment could be checkpointed. Five retries, all inside the same unattended run. Nothing was repaired after it.

First attempts fail in AI coding, and in human coding. The useful question is whether the workflow notices a failure, contains it, repairs it, validates again, and records what happened, instead of quietly accepting a broken step. In this run it did, and the history shows each step. Repair is not a weakness of the case study; it is one of the clearer signs that the workflow behaved as intended.

Verification was automated, and deepest on the logic

The app was not declared complete because it looked right in a screenshot. CodeLead required the build, the tests, the browser check and the rule check to pass before it marked work complete. No person was testing routes or deciding when to continue. That is the difference between a model generating code and a system delivering a verified result.

The checks were not equally deep. The build and the full unit-test suite passed at every checkpoint, and the business rules in the request were checked against the finished application by a 30-case oracle independent of the model's tests. The browser check was thinner: of the 18 acceptance criteria it audited, 11 were readable by a browser; 6 passed, 1 failed and was repaired, and the rest could not be observed by a browser at all, because a pure-logic module renders nothing to click.

A capability the browser could not observe never counted as a pass; those rested on the build, the tests and the rule check, and the ledger above records them that way. The business rules were verified more deeply than the interface, which is why the interface is the part we ask you to judge for yourself.

This was not the first run

The pipeline was changed one thing at a time between runs of this challenge, each run scored the same way, and the run on this page is the most recent on the current configuration. The earlier published run used the same model and prompt served by LM Studio, at about 18 tokens per second; this one is served by TensorFold with speculative decoding, which is where most of the time went. The development history is in the case-study repository.

Governance made the result possible

The control arm is the cleanest evidence. The same model, raw, finished a good-looking application and got a business rule wrong without any way to know. Under CodeLead the same model's failures were caught at the gate and repaired before they could land, and its business rules passed an independent check, 30 of 30. The model did not get smarter. The work got more disciplined.

The same gates ran at every model size. A 4-billion-parameter model verified 4 of 14 increments and stopped, with every unbuilt increment recorded as not built: a smaller application that works, rather than a broken one. The smaller the model, the more the workflow has to carry, and the more the checks matter.

This page describes the method at the level of the stages above and no further; the underlying methods are the subject of provisional patent applications. The operating principle is the point. Capability does not come only from model size. It also comes from how the work is framed, bounded, checked, repaired, and approved.

What people approved

During this benchmark run, nothing was approved mid-run, and that was deliberate: the point was to see how far the governed loop could get unattended after receiving the same challenge prompt. A person chose the task and the declared stack change, and reviewed the result afterwards. No person wrote or edited application code.

A modernization engagement works differently. There, people approve the goal, the plan, the scope of each change, and anything risky before it lands.

What evidence was captured

Every verified increment was committed with its evidence: what was asked, what changed, and what the checks concluded. For this run that is the run journal, the console and serving logs, the session log with its 48 model calls, and the oracle results, all being prepared for publication in the case-study repository. The application repository holds the git history of the 2026-09-14 run, one commit per verified stage, with its evidence ledger and session summaries. The case-study repository holds the control arms, the other-models record, the two off-domain requests, and the reproduction materials. Model input and output transcripts are withheld.

What this does not prove

Why it matters

Many teams assume the choice is binary: a small local model with weak results, or a frontier cloud agent with the cost, the privacy exposure, and the unpredictable usage.

This run points to a third option. A fixed-cost local machine, a medium model, and a disciplined workflow produced useful, validated software without the codebase going to a cloud agent, and with every claim about the result checked by a machine rather than by the model that made it.

Side by side, the three results say different things. The frontier cloud agent was faster and more polished. Kimi K3 showed what a very large model can do on a large local setup. This run suggests that a much smaller model on one Mac can complete meaningful, verified engineering work when the workflow supplies the missing discipline: planning, a bounded build, repair at the gate, independent verification, evidence and guardrails, automated after the prompt.

So local does not have to mean a four-machine cluster or frontier-scale hardware, and “it compiles and the tests pass” is not the same as “it charges the right amount”. The claim is not that smaller models beat larger ones. It is that orchestration changes the economics, because a smaller model can carry more of the work when the system around it does the rest.

The same discipline is what a legacy system needs: understand the code before changing it, keep people in charge of the plan, validate every step, and leave evidence for the next maintainer.

Reference comparison from Alex Ziskind's video, sponsored by Abacus AI; the Kimi K3 and Abacus figures are as stated in the video. CodeLead run: 2026-09-29, qwen3.8-27b (8-bit) served by TensorFold with speculative decoding on a MacBook Pro with an M5 Max and 64 GB. Run figures: the run journal, console log, serving log and oracle results of run k3-tensorfold-2026-09-29a.