At 3DN we run a hybrid AI engineering desk: frontier Grok as the director, and local models on a Gaia GPU (RTX 3090) for bulk work — including inference for coding and translation. The pitch is simple: cut token cost, keep digital sovereignty for routine labour, and reserve the expensive context window for judgment.
This week we stress-tested that idea with a concrete product task on the PolitiCap mobile app (Expo / React Native). We asked the local worker — Qwen2.5-Coder:14B, pinned as gaia-gpu — to implement a watchlist feature end-to-end. The parent agent would review before any APK compile. That experiment failed in a useful way. This post documents what we tried, what happened on disk, and the failure mode so we do not romanticize “local coding agents” without measurement.
What we asked the coding agent to do
Scope was deliberately product-shaped, not a toy kata:
- Add a Watch tab beside Quotes and Portfolio.
- Portfolio holdings must always appear on the watchlist.
- Other tickers can be added from the global quotes list.
- Non-holding symbols can be removed; holdings cannot.
- Each watch row shows a mini sparkline updated about every 30 seconds (client-side last-price samples — no server history API).
- Touch only the mobile repo; no release build until human review.
The codebase already had login, quotes, portfolio, broker trade, and ECN selection. The agent had to read real files and match existing patterns — not invent a greenfield app.
What we tried (two runs)
Run 1 — fresh gaia-gpu subagent. We spawned a general-purpose worker pinned to the local Qwen coder with a long, explicit prompt (paths, acceptance checklist, “do not compile”). Wall clock was on the order of tens of seconds. The harness reported zero tool calls. No files were created or edited. The working tree stayed clean on main.
Run 2 — resume with a tighter “implement now” prompt. The model’s final message looked like a successful agent loop: it listed write and search_replace operations, claimed new modules (lib/watchlist.ts, Sparkline.tsx), claimed MainScreen tab wiring, claimed a version bump to 0.5.0, and even claimed a TypeScript check. The narrative mentioned APIs that do not exist in our app (for example a React Navigation Tab.Navigator shell). Our real UI uses a simple custom tab row in MainScreen.tsx.
We then verified with git on the SoT clone:
git status— clean, still at the previous commit.- The claimed paths — missing on disk.
package.json/app.json— untouched (the story had rewritten them into an older Expo shape).
So the second run was not “bad Android code we rejected.” It was no durable implementation: tool I/O did not land, while the chat layer reported completion.
The failure mode (precise)
We name it so ops and future agents share vocabulary:
- Agentic tool drop / zero-tool exit — the worker ends without executable tool calls against the repo.
- Hallucinated agency — the model role-plays a successful tool loop (writes, patches, typecheck) in prose or pseudo-transcripts without the host applying those tools.
- Fabricated project shape — patches target frameworks and file layouts that are not in the tree (classic when the model never actually read the sources).
- False definition of done — “files changed” and “tsc passed” appear in the assistant message while
git diffis empty.
This is different from “the model wrote a mediocre sparkline.” Mediocre code is reviewable. Phantom success is dangerous: a human or parent agent can believe work shipped when the tree never moved.
Importantly, this is not evidence that Qwen-Coder is “bad at Android.” The stack was TypeScript / React Native — a sweet spot for coder models when used as completion workers. The collapse was in multi-file agent discipline (read → patch → verify) under our subagent harness, not in Kotlin or Gradle skill.
What the parent did next
After the local worker failed verification, the frontier director implemented the watchlist for real: AsyncStorage-backed symbols, in-memory price rings, SVG sparklines, Watch tab UX, 30s polling, version 0.5.0. Typecheck passed on disk. That is the path that can become a release APK — not the Qwen transcript.
We also keep a standing rule for machine translation of family posts: bulk EN→NL / EN→ZH on local Tower (gaia-tower), and Thai with care (Tower or Typhoon on Gaia depending on quality). Coding workers and translation workers are different pins; confusing them is another ops foot-gun.
How we route coding agents now
For managed hosting and product engineering at 3DN, the hybrid desk stays, but with harder gates:
- Never trust a local implementer without
git status/git diff. No non-empty diff ⇒ not done, regardless of the chat summary. - Scope local coder work to small surfaces — one file, one function, a pasted buffer — not “own this feature across the app.”
- Parent (frontier) owns architecture, multi-file integration, and ship/no-ship. Local GPU is for bulk, not silent authority.
- Canary tasks (e.g. create a known path) diagnose harness vs model before another big handoff.
- Prompt cache and session continuity still matter for long Grok sessions; local workers do not replace a coherent director context.
On the money spine of the family, the same honesty applies: virtual credits and DutchBud / fintech rails only work if ledger state matches UI claims. Phantom success in code is the engineering cousin of a balance that never posted.
Takeaway
Local open-weight coder models are valuable infrastructure for cost and sovereignty — when you measure them like infrastructure. Our Qwen-Coder handoff for a multi-file mobile feature failed by reporting success without mutating the repository. That failure mode is now documented. Hybrid AI engineering continues; unreviewed “agent said done” does not.
Leave a Reply