Teaching Tower Thai: our plan for a TowerInstruct LoRA

At 3DN we already run local inference on a local workstation GPU (RTX 3090) for family-site machine translation and for hybrid AI engineering. Bulk EN→NL and EN→ZH drafts go through Tower on Ollama; Thai has been the weak link, so we lean on Typhoon for TH today. The next step is not “buy more API” — it is a deliberate LoRA experiment on the same infrastructure, under our own roof, for digital sovereignty and lower long-run token cost.

This post is the public plan: what we train, why that base model, how we keep production safe, and where Unbabel’s work fits in. Snippets below are the shape of the lab — not a paste-and-forget cookbook.

Why a LoRA at all?

Full fine-tunes of multi-billion-parameter translation models are expensive and easy to overfit. A LoRA (low-rank adapter) trains a thin set of matrices on top of a frozen base. For a desk that already hosts Tower weights, that is the right learning curve: measurable EN→TH gains, a Hugging Face adapter we can publish, and an optional merge later if we want a dedicated Ollama pin.

We are not training the GGUF blob that Ollama serves day-to-day. Training uses Hugging Face / PEFT-style weights; serving can stay on the current Tower-Plus Q4 path until eval says otherwise.

# Mental model: freeze the base, train thin adapters
# ΔW ≈ B @ A  with rank r << d_model
# Only A, B (and maybe biases) get gradients.

from peft import LoraConfig, get_peft_model, TaskType

lora = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],  # start small; expand if underfit
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)
# model = get_peft_model(base_model, lora)
# model.print_trainable_parameters()  # expect ~0.1–1% trainable

Base model: TowerInstruct-7B v0.2

Community guidance (and our own read) points at starting from TowerInstruct, not a generic chat LLM. The family is built for translation-shaped prompts — the same pattern we already use on the desk.

We are installing Unbabel/TowerInstruct-7B-v0.2 as the first train base:

7B fits QLoRA comfortably on a desktop 3090 with room to iterate.
v0.2 over v0.1 for the cleaner instruct/MT behaviour.
13B or Tower-Plus-9B only after the 7B recipe works — more capacity, slower loops.

Unbabel’s Tower line lives at unbabel.com. Public weights: Hugging Face Unbabel/TowerInstruct-7B-v0.2.

# Lab layout on a local workstation (HF weights, not Ollama GGUF)
# …/tower-lora/
#   models\TowerInstruct-7B-v0.2\   # safetensors shards
#   datasets\en-th\                 # jsonl pairs
#   adapters\en-th-r16\             # PEFT output
#   out\

# Stage download on a build host, then LAN copy (multi-GB, offline train later):
python -c "from huggingface_hub import snapshot_download; \
snapshot_download('Unbabel/TowerInstruct-7B-v0.2', \
local_dir='/var/tmp/hf-models/TowerInstruct-7B-v0.2')"

Data shape (programmers care about schemas)

One row, one job. Match the prompt style we already use in production MT so train ≈ serve:

{
  "instruction": "Translate the following English source text to Thai.",
  "input": "Every voter deserves a market.",
  "output": "ผู้มีสิทธิเลือกตั้งทุกคนสมควรได้รับตลาด"
}

# Tiny loader sketch — holdout is sacred
import json
from pathlib import Path

def load_jsonl(path: Path):
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if line.strip():
            rows.append(json.loads(line))
    return rows

def split_holdout(rows, frac=0.1, seed=7):
    import random
    rng = random.Random(seed)
    idx = list(range(len(rows)))
    rng.shuffle(idx)
    n = max(1, int(len(rows) * frac))
    hold = {idx[i] for i in range(n)}
    train = [rows[i] for i in idx if i not in hold]
    test  = [rows[i] for i in idx if i in hold]
    return train, test

# Prefer: our EN desk copy + Typhoon draft + light human gold edit
# Avoid: unedited bulk MT soup, HTML debris, mixed dual-language lines

How we will approach the work

Install the HF base on a local workstation — full safetensors tree under …/tower-lora/models\ (download staged on our build host, then LAN copy).
Data before rank — a few hundred to a couple of thousand clean EN→TH pairs; hold out ~10% forever.
Train QLoRA — rank 8–16, short context, one heavy job on the 3090 at a time (unload competing Ollama models).
Eval honestly — base Tower vs LoRA (and later Typhoon); chrF/BLEU as a guide, humans for names and product terms. First holdout table is above.
Publish when it wins — Hugging Face adapter card; only then merge→GGUF / optional local workstation-tower-th pin.

# QLoRA load sketch (4-bit base + LoRA) — 3090-friendly
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

base_id = r"…/tower-lora/models\TowerInstruct-7B-v0.2"
tok = AutoTokenizer.from_pretrained(base_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    base_id,
    quantization_config=bnb,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
# then get_peft_model(model, lora_config) and Trainer / custom loop

# Production MT prompt shape we already use (train should rhyme)
def tower_prompt(en: str, lang_name: str = "Thai") -> str:
    return (
        f"Translate the following English source text to {lang_name}.\n"
        f"English: {en.strip()}\n"
        f"{lang_name}:\n"
    )

# Desk rule until LoRA wins holdout:
#   NL/ZH  -> local workstation-tower (Tower-Plus)
#   TH     -> local Typhoon-translate
#   judgment / SEO -> frontier Grok (not bulk MT)

# Ship gate: never trust vibes
def better_than_baseline(scores: dict) -> bool:
    # scores = {"tower": chrF, "lora": chrF, "typhoon": chrF}
    return scores["lora"] > scores["tower"] + 0.5  # margin TBD on real holdout

# if not better_than_baseline(...): keep Typhoon in prod; LoRA stays lab

Production MT policy stays dual-track until eval says otherwise: Tower for NL/ZH, Typhoon for TH, frontier Grok for judgment — not for bulk translation.

What success looks like

A reproducible train script and dataset schema on a local workstation.
An adapter that beats stock TowerInstruct on our Thai holdout without wrecking pairs we add later.
A Hugging Face card others (and future us) can load with PEFT on the same base.
No romance: if Typhoon still wins, TH production stays on Typhoon.

# After train — what we expect on disk
# adapters/en-th-r16/
#   adapter_config.json
#   adapter_model.safetensors
#   README.md   # base id, pair, r, data size, license, eval table

Compute and ops

This is managed hosting and lab compute on metal we control — the same philosophy as hybrid coding agents and local Flux thumbs. Long sessions still benefit from careful prompt cache and session continuity on the director model; the LoRA job itself is a bounded GPU batch, not an always-on API.

On the family money spine, the same honesty applies: virtual credits and DutchBud / fintech rails only work when state matches claims. A translation adapter only “ships” when eval matches the card.

# One heavy job on the 3090 — do not co-host train + Flux + Ollama-14B
# (ops checklist, not poetry)
# 1) stop competing GPU consumers
# 2) train QLoRA
# 3) write adapter
# 4) free VRAM, restore Ollama desk pins

Status — first lab numbers (smoke, not production)

The first QLoRA adapter is on disk. Train set: 110 curated EN→TH pairs; holdout: 20 unseen lines. Recipe: TowerInstruct-7B-v0.2 base, rank r=16 on q/k/v/o, ChatML prompts, 3 epochs, train loss ≈ 1.77 (~3.5 minutes after weight load on a single desktop 3090).

Holdout automatic scores (sacrebleu corpus BLEU / chrF, greedy decode, EOS stop):

Model	BLEU	chrF
Base TowerInstruct-7B-v0.2 (4-bit)	2.06	14.54
Base + EN→TH LoRA r16	2.88	16.22
Delta	+0.82	+1.68

Read this cold: the adapter beats stock Tower on this tiny holdout by a small margin. Absolute scores are still low — the base was weak on our desk Thai, and 110 rows is a pipeline proof, not a production corpus. We are not flipping family-site Thai MT off Typhoon on these numbers. The ship gate from the plan still holds: LoRA only replaces Typhoon when holdout + human review say so on a larger gold set.

What we did prove: QLoRA train → PEFT adapter save → base-vs-adapter eval with BLEU/chrF on the same workstation stack, offline after the HF weight tree is local. Next lab steps are more EN→TH gold (and careful Typhoon-draft + edit), not romantic claims.

Further reading: Unbabel · TowerInstruct-7B-v0.2 on Hugging Face.