Frontier models fail three operational tests at once: they do not take new skills without risking the old ones, they have no night shift, and the serving weights are the only copy. Incremental learning, plasticity, and maintenance are one problem with three names. The usual answer is a bigger river — more HBM, more tokens, same mesh.
3DN wants a delta.
Language is the bus — tokens are the currency
An LLM’s job is not only to “speak.” After pretraining it already has a shared token space, next-token control, and a pile of world patterns learned through text. Language is the training medium and the I/O format. Internally the net never sees English letters; it sees token IDs → embeddings → attention → next-ID prediction.
That is why modular systems still start from a trained backbone: you inherit tokenizer, embedding space, and weights that already map human text into a usable vector language. You hang modules on that bus. You do not invent English from a pile of tiny specialists and hope they invent grammar later.
But a router is a different job. A dispatcher whose only output is “which expert gets this” should not chat. Natural language is the wrong control plane for that hop.
Leaves, not a soup
A specialist can be ~19k parameters and twenty seconds on a single GPU if the job is fixed and dense — for example an 8×8 glyph classifier. That net should never see English first. Sending sixty-four floats through a chat model so it can say “that looks like an A” is slower, vaguer, and worse.
Train the cheap ones from scratch. Reuse the expensive ones. Keep specialist I/O in tensors, class IDs, or a tiny codebook. Use a language model only where the output must be language.
| Job | Right module size |
|---|---|
| 8×8 glyph | 10k–100k |
| Which expert? (router) | 10k–few million on an embedding |
| Retrieve similar docs | embedding model, not a chat model |
| Write an explanation | LLM |
| Run the code | interpreter, not a net |
MoE inside a model ≠ modular systems
In current LLM talk, mixture of experts usually means sparse specialist FFNs inside one Transformer: same shape, token-level gate, shared residual stream (Mixtral / Switch / DeepSeek-style). Industry slang drifted from older “gate + heterogeneous experts.”
What we mean for infrastructure and AI engineering desks is closer to macro-experts: separate networks or models, request-level routing, specialists you can train offline without touching the writer. You can run both: outer router picks subsystem; inner MoE picks which FFNs fire per token.
The router is the unit you can still teach
A small gate is cheap to rebuild, not only cheap to run. Replay a dispatch log, expand the output head with a new class, retrain in minutes — without touching specialists. Freeze experts; only the router (and maybe a new expert) gets gradients. New problem type = new expert index. Old indices stay stable. Uncertain → fallback generalist, then label the miss in sleep.
Repeat the gate where a boundary pays: task family → skill → version → physics. The same object stacks:
request → SRAM router # finish in cache / this die → HBM router # local GPU working set → board router # other GPU on the node → fabric router # other node → region router # other facility / residency / cost
Semantics says what should run. Tier says where it is allowed to run. That is NUMA-aware scheduling with a training signal: wrong path + latency + bytes moved + energy + residency. Most tokens never need the flood-stage generalist.
If every tiny op is a learned router you pay hop error and latency. Use a router when the next piece is separately trainable, worth isolating against forgetting, and skippable for most inputs. Don’t route a cheap always-on transform.
Sleep is HA, not poetry
Dense frontier serving weights are the knowledge. There is no spare lobe you can take offline. Modular routing makes a maintenance window possible:
- Hot twin serves.
- Cold twin replays, distills, prunes, evaluates.
- Atomic swap: cold becomes hot.
- The rest of the DAG stays up.
That is high availability plus consolidation — the engineering cousin of a dream cycle. Put twins on small high-churn modules (routers, classifiers, tools). Keep one copy of the huge writer until you can distill it. A 20-second retrain on a 3090-class box is a realistic sleep cycle; a full frontier retrain is not.
This does not create biological plasticity inside a frozen API leaf. It sidesteps the failure mode of “one net, one lifelong gradient.” Plasticity ≈ we can still add capacity. Incremental learning ≈ we can add a skill without a full retrain. Sleep ≈ we can rewrite a module offline.
Memory wall: river vs delta
Large LLM inference is often memory-bandwidth bound: pull a huge weight slab from HBM, do a cheap matmul, pull the next slab. A mesh of small nets wins when hot experts pin next to the ALU and only the path’s weights move. The wall moves if hops get chatty across the fabric — then you trade the weight wall for the network wall.
Tiered routers make distance visible. Traffic that avoids streaming a 70B slab is a feature, not a bug — a controlled ascent up the hierarchy. One river becomes a braided delta: most water never leaves the near channels; only flood-stage problems take the deep channel to the generalist.
Deltas silt up (dead experts, gates that always promote, load collapse onto the default route). Sleep and replay are dredging. Hierarchy and pinning are levees. Fallback to the main channel is the floodplain you keep on purpose.
The internet rhyme
Packet networks already solved “vast types, distance is expensive, parts go down, policy at the edge.” Hierarchical routers, local default, promote when you must, intelligence at the leaves — that is BGP and CDNs with different nouns. Your SRAM/HBM/board/region gates are routing on a ticket (type, budget, residency). Specialists are servers.
Plain forwarding stays the default. Most packets should mean: move these bits, don’t interpret them. Transformation is opt-in on declared waypoints (opaque | reduce | policy | enrich) with receipts. If the mill is asleep or untrusted, the canal still carries bytes. An internet that can only transform is a middlebox. An internet that can forward and sometimes transform is a delta with a bypass around every mill.
What this does not solve
- Composition still needs an orchestrator; hybrids are pipelines, not one mega-enum of types.
- Shared facts still need retrieval / memory, or they fork across leaves.
- The open inventory of problem types is discovered (cluster residuals in sleep), not fully designed on day one. Hierarchy keeps each gate small.
- The slush — tasks that will not sit in a type — is what you still pay a frontier generalist (or a human) for. The architecture’s job is to make that slush smaller over time.
How 3DN already runs the incomplete version
This is not only a drawing. On the 3DN family desks we already route work as a hybrid fabric: frontier Grok for judgment and long sessions, local GPU workers for bulk inference and translation, adapters trained offline so the serving base stays frozen, sealed paths that must not rewrite every hop. DutchBud bank virtual credits and the PolitiCap civic market are family products on that same spine — work · money · politics — with dibs as closed-loop play credits, not a casino cash-out and not a licensed open-banking claim.
PolitiCap’s public tape is an early macro-router in the wild: named destinations (tickers), append-only receipts (comments and fills), optional second hops when first-pass sides conflict. WordPress is a terrible tensor fabric and a decent off-board bus for political capital. The same rule applies: forward by default; debate is an opt-in waypoint with a hop budget — not five full salons on every headline.
The article is the drawing. The desks are the first creeks. Build the delta: small experts, hot/cold sleep, tiered distance, opaque forward as default. Don’t invent language from modules. Invent modules on top of language — and put a disposable gate in front of every expensive path.
Leave a Reply