Abstain or Route

Route it, or hand it off.

Intent routing across 10 languages, with LoRA written from scratch.

Read the code The findings
Test accuracy, %MASSIVE 1.1, seed 0
Runmean of 10enhitasw
Probe, en55.778.866.617.716.5
LoRA, en67.087.174.947.923.8
Full, en66.888.175.439.324.1
Probe, multi58.475.166.831.322.8
LoRA, multi76.983.979.073.358.7
Full, multi79.986.182.177.165.6

Highlighted: LoRA. Full fine-tuning trains 127× its parameters for 3.0 more points.

Four findings

  1. LoRA ties full fine-tuning on a narrow target

    Trained on English, the mean gap is +0.2 points [−0.2, +0.6]. Across ten languages, rank 8 trails by 3.0.

  2. English-only training fails where pretraining is thin

    Swahili falls to 23.8% and Tamil to 47.9%. The same budget over ten languages lifts them to 58.7% and 73.3%.

  3. Calibration breaks in the same languages

    Expected calibration error for Tamil is 0.120 after English-only training, 0.028 after multilingual.

  4. Handing off cannot rescue an unknown language

    Deferring the least-confident 20% buys Hindi 10 points and Swahili 4.

Confidence only helps where it separates right from wrong

LoRA, English-trained: accuracy on kept requests.

Accuracy on the requests the model keeps, as more are handed to a human Handing off 20% lifts Hindi from 74.9% to 84.8% and English from 87.1% to 94.8%; Swahili only from 23.8% to 28.1%, Tamil from 47.9% to 55.9%. 0 25 50 75 100 0% 10% 20% 30% 40% 50% Share handed to a human Accuracy on kept, % English 98.5 94.8 Tamil 69.7 55.9 Swahili 37.7 28.1 Hindi 94.9 84.8 Accuracy on the requests the model keeps, as more are handed to a human
Rule at 20%; end labels at 50% handed off.

Three ways to train

One encoder, paraphrase-multilingual-MiniLM-L12-v2, plus a linear head. Apple M2, 8 GB.

MethodWhat trainsParameters
ProbeThe head only23,100
LoRA r = 8Query and value, α = 16, and the head170,556
FullEvery layer and the head; word embeddings frozen21,662,652

Limits

  • One seed per run. The confidence intervals cover test sampling, not training variance.
  • Full fine-tuning froze the word embeddings: 96M of 118M parameters.
  • One LoRA configuration; no rank sweep.
  • Why Swahili is weakest is untested.

Reproduce

Saved logits in results/ rebuild every table in seconds.

git clone https://github.com/akshit9162/abstain-or-route
cd abstain-or-route
python -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/python -m src.evaluate   # every table
.venv/bin/pytest                   # 12 after setup
PATH=.venv/bin:$PATH ./setup.sh    # data and encoder
rm results/*/meta.json && ./run.sh # ~2 h on an M2