tr-telekom-router

A 0.8B model that routes Turkish telecom support messages in 64 ms, and knows when to call a human.

Most routing models give you a label. tr-telekom-router gives you a decision you can trust: a probability for each of six teams and a calibrated confidence. Above the threshold it routes the message instantly; below it, the message goes to a person. On a sealed test set it never saw during development, it routed 84.4% of messages on its own with 98.7% accuracy and passed the rest to a human.

93.3% accuracy on the sealed holdout	up from 55.6% for the same model before training
98.7% accuracy on the messages it routes itself	at 84.4% automatic coverage
64 ms p95 latency per message	one forward pass, one token, 0.8B parameters
97% answers stable when the options are shuffled	up from 41%: position bias trained out

What it means for a support team

Picture 100 incoming messages. In our sealed test, the model sent about 84 of them straight to the right team, with roughly one mistake among those 84, and flagged the remaining 16 for a person to read. Agents stop sorting the easy majority and spend their time on the messages that genuinely need judgement. Because the model is small, it answers in a few hundredths of a second and runs on a single modest GPU, so it can sit in front of every chat, app form and transcribed call without becoming the bottleneck or the main cost.

Türkçe

Türkçe telekom müşteri mesajlarını 64 milisaniyede doğru ekibe yönlendiren, emin olmadığında insana devreden 0,8 milyar parametrelik bir karar modeli. Altı ekip için ayrı olasılık ve kalibre edilmiş bir güven değeri verir. Geliştirme sırasında hiç görmediği mühürlü test setinde mesajların %84,4'ünü kendi başına yönlendirdi ve bunların %98,7'sini doğru ekibe gönderdi; kalanını insana devretti. Eğitimden önce aynı modelin doğruluğu %55,6 idi, eğitimden sonra %93,3.

Yöntemin özü: büyük bir öğretmen modele her müşteri mesajı 8 kez, seçenekler her seferinde farklı sırada soruldu. Oylar "bu mesaj %75 iptal, %25 fatura" gibi bir olasılık dağılımına dönüştü ve küçük model bu dağılımı taklit etmeyi öğrendi (bilgi damıtma, knowledge distillation). Seçenek sırasını karıştırmak öğretmenin konum yanlılığını sildi; model de seçeneğin yerine değil içeriğine bakmayı öğrendi. Doğru puanlama kuralına dayalı kayıp (KL + Brier), eğitim sonrası sıcaklık ayarı ve ayrı bir sette seçilen eşik, modelin ne zaman emin olmadığını bilmesini sağlıyor.

Why it is new

Routing is usually solved one of two ways: a text classifier that outputs a label and an uncalibrated score, or a large language model that is slow, expensive and has to parse its own free text. tr-telekom-router takes a third path, and combines four ideas that, to our knowledge, have not been put together for Turkish customer support before.

1. A language model that decides instead of talking. The model keeps the language understanding of an instruction-tuned LLM, yet answers like a classifier: it reads the probability of six option letters at one position and stops. No classification head is bolted on, no output is parsed, no label can be hallucinated, and every decision costs one prefill and one token. The checkpoint stays a standard causal LM that vLLM and transformers serve unchanged.

2. Order-shuffled distribution distillation. Large models have a favourite answer slot, and a student copies it. Instead of asking the teacher once, we asked it eight times, with the options in eight different orders, and turned the votes into a probability distribution. The teacher's position bias cancels out, and the student learns content, not position. The effect is measurable: with options shuffled, the untuned model kept its answer 41% of the time; this model keeps it 97% of the time.

3. Label quality without a single human label. One model family wrote the training messages, a second, independent family labelled them, and a message survived only when both agreed. Cross-family agreement replaces the annotation team in a domain where no open Turkish dataset has these classes.

4. Calibration built into the objective, not bolted on. The model is trained on a proper scoring rule (KL divergence plus Brier score), then a temperature is fitted on held-out data and a hand-over threshold is chosen on a separate split. The result is a single dial: move the threshold and trade automation for accuracy along a known curve.

And it was measured like a production system, not a demo. The final test set was written by a separate process, hash-sealed before training and opened exactly once; the temperature and threshold were frozen before it was opened. Every number on this page comes from that single measurement or from the development split, with paired-bootstrap confidence intervals over message families.

In plain words

The ideas behind this model, without jargon. Each term links what the page says to what it means.

Term	What it means here
Decision model	A language model used to pick one of a fixed list of answers and say how sure it is, instead of writing free text.
Teacher and student	The teacher is a large, expensive model that already does the task well. The student is this small model, trained to copy the teacher's judgement so it can run fast and cheaply.
Knowledge distillation	That copying process. The student learns from the teacher's answers rather than from human-written labels.
Soft label	Instead of "this message is billing", the target says "75% cancellation, 25% billing". It tells the student how clear-cut each example is.
Position bias	The habit of a model to prefer an answer because of where it sits in the list (say, always option A), not because of what it says.
Token, logit, softmax	A token is a piece of text the model reads or writes, here a single letter A–F. A logit is the model's raw score for a token. Softmax turns the six scores into six probabilities that add up to 100%.
Full fine-tune	Continuing to train every weight of an existing model on new data, as opposed to training a small add-on.
Loss	The number training tries to make smaller: how far the model's probabilities are from the target. The training-loss chart shows it falling.
KL divergence	A standard measure of how different two probability distributions are. Zero means identical.
Brier score	The squared distance between predicted probabilities and the truth. It punishes being confidently wrong and is bounded, so a few odd examples cannot dominate.
Proper scoring rule	A scoring method under which the best strategy is to report your honest probability. KL and Brier are both proper.
Calibration	Whether the stated confidence matches reality: of all the answers given with 80% confidence, about 80% should be right.
Temperature	A single number applied after training that softens (or sharpens) every probability, used to fix over- or under-confidence.
Threshold, coverage, hand-over	The threshold is the minimum confidence for an automatic decision. Coverage is the share of messages above it. Everything below is handed over to a human agent.
Holdout	Test messages locked away before training and used exactly once at the end, so nobody can tune the model toward them.
95% confidence interval	The range the true improvement most likely lies in, given the size of the test set. Here: between +26.7 and +48.9 points.
p95 latency	95% of messages get their answer within this time.

The data

No real customer data was used. The training set was built from scratch in four steps.

flowchart TD
  W["Writer model<br/>120 structured prompts → 1,200 messages<br/>6 teams × 10 subtopics × mix of 10 styles"] --> L
  L["Teacher model, different family<br/>each message 8 times, options in 8 orders<br/>9,600 answers"] --> V["8 votes → probability distribution per message"]
  V --> F["Keep only where writer intent and teacher majority agree<br/>and at least 6 votes are readable · 1,155 messages"]
  F --> C["Render with the student's chat template<br/>2 option orders per message · 2,076 training / 234 validation rows"]

Writing. Each of 120 prompts asked a writer model for ten distinct messages for one team, one subtopic and a mix of three styles drawn from ten: no Turkish characters, formal elderly customers, youth slang, voice-to-text without punctuation, all-caps anger, regional speech, English words mixed in, app complaint forms, one-liners and long multi-sentence accounts. Seven of every twenty prompts asked for multi-team messages that also mention a named second team, so the model learns the precedence rules that decide them. The subtopics cover the real support surface: bill disputes, auto-payment, refunds and late fees for fatura; porting in, porting out, rejected or stalled porting for numara_tasima; SIM loss, eSIM transfer, installments and warranty for cihaz, and so on for every team.

Labelling. Each message went to the teacher eight times at temperature 1.0 with the six options re-ordered every time; each vote is mapped back from its letter to its team. The eight votes form the target distribution.

Filtering. 37 messages were dropped because the teacher's majority disagreed with the team the writer intended, 8 because fewer than six votes were readable. Every message was also checked against every evaluation split for exact and near duplicates (similarity ≥ 0.85): none found.

Property of the 1,155 training messages	Value
Messages per team	189 – 195 (balanced)
Single-team / multi-team	765 / 390 (34% multi-team)
Length in words, median (5th – 95th percentile)	8 (3 – 20), longest 49
Written without Turkish characters (ç ğ ı ö ş ü)	21%
Containing a fully capitalised word	12%
Phrased as a question	17%
Unanimous teacher votes	96.4%
Mean top probability of the target	0.991
Readable votes per message: 8 / 7 / 6	990 / 134 / 31 messages
Prompt length after the chat template, median (max)	818 (891) tokens

Training messages and the targets the model learned from; the soft ones are where the teacher was split:

Message	Target distribution
asker tarifem iptal	iptal_sikayet 0.75 · tarife_paket 0.25
hat gecisinde askida fatura var	numara_tasima 0.625 · fatura 0.375
telefon kurulum takildi internet yok	cihaz 0.75 · teknik_ariza 0.25
aramada 45g ye dusuyor	teknik_ariza 1.0
eski operatorden ayrılıp size geliyorum hat acilmi	numara_tasima 1.0
Kart kayboldu, yeni kart basıp yüksek tarifeye geçmek istiyorum.	cihaz 1.0
85 yaşındayım, verdiğiniz tablet arızalandı, iadesi için dilekçemi yazıyorum, aksi halde tüketici hakemine giderim.	iptal_sikayet 1.0

The mathematics

You do not need the formulas to use the model. Each step below ends with a one-line summary in plain words.

Readout. For a message $x$ and an ordering $\pi$ of the six options, the prompt ends with the opened answer Cevap: (. Let $z_\ell$ be the model's logit for the token of letter $\ell \in {A,\dots,F}$ at the next position. The probability of option $j$, shown at letter $\pi(j)$, is a softmax over the six letters only:

$$p_{\theta }(j\mid x,\pi )=\frac{\exp z_{\pi (j)}}{\sum _{\mathrm{\ell }=A}^F\exp z_{\mathrm{\ell }}}p\_\backslash theta(j\; \backslash mid\; x,\; \backslash pi)\; =\; \backslash frac\{\backslash exp\; z\_\{\backslash pi(j)\}\}\{\backslash sum\_\{\backslash ell=A\}^\{F\}\; \backslash exp\; z\_\backslash ell\}$$

In plain words: the model looks at the six letters it could write next and turns their scores into six percentages. It never writes anything else.

Target. The teacher is asked $K = 8$ times with orderings $\pi_1,\dots,\pi_K$. Each vote $v_k$ is mapped back to an option id, and the target is the vote frequency:

$$t_j=\frac{1}{K}\sum _{k=1}^K\mathbb{1}[v_k=j]t\_j\; =\; \backslash frac\{1\}\{K\}\; \backslash sum\_\{k=1\}^\{K\}\; \backslash mathbb\{1\}\backslash left[v\_k\; =\; j\backslash right]$$

Because every $\pi_k$ is different, a teacher that prefers a letter regardless of content spreads that preference over all options instead of onto one, so the bias averages out of $t$.

In plain words: if the teacher picked "cancellation" six times and "billing" twice out of eight, the target is 75% cancellation, 25% billing. Shuffling the options each time stops the teacher's favourite slot from leaking in.

Objective. Each training row is a message rendered in one of two orderings. The loss at the answer position is the KL divergence to the target plus a Brier term with weight $\lambda = 1$:

$$\mathcal{L}(\theta )=\underset{\mathrm{K}\mathrm{L}(t\mathrm{\parallel }p)}{\underbrace{\sum _j{t}_j\log \frac{t_j}{p_j}}}+\lambda \sum _j{(p_j-t_j)}^2\backslash mathcal\{L\}(\backslash theta)\; =\; \backslash underbrace\{\backslash sum\_\{j\}\; t\_j\; \backslash log\; \backslash frac\{t\_j\}\{p\_j\}\}\_\{\backslash mathrm\{KL\}(t\; \backslash ,\backslash |\backslash ,\; p)\}\; +\; \backslash lambda\; \backslash sum\_\{j\}\; \backslash left(p\_j\; -\; t\_j\backslash right)^2$$

Both terms are strictly proper: each is minimised only at $p = t$, so the model is rewarded for reporting the teacher's uncertainty, not for sharpening it. The gradient of the KL term with respect to the logits is $p - t$, which pulls the whole distribution toward the target. The Brier term is bounded (at most 2 per row), so a few confidently wrong rows cannot dominate the update the way the unbounded log term can. Training on two orderings of each message makes the learned mapping invariant to where an option is placed.

In plain words: training pushes the student's percentages toward the teacher's percentages, and charges extra when it is confidently wrong. Seeing each message with the options in two different orders teaches it that the letter does not matter, only the content.

Temperature. Since 96% of the targets are unanimous, the trained model is over-confident. One temperature $T$ rescales the readout probabilities and is fitted by minimising the negative log-likelihood of the correct team on a separate calibration split $\mathcal{C}$ of 60 messages:

$$p_j^{(T)}=\frac{p_j^{1\mathrm{/}T}}{\sum _i{p}_i^{1\mathrm{/}T}},T^{\star }=\arg \underset{T}{\min }-\sum _{(x,y)\in \mathcal{C}}\log p_y^{(T)}(x)=2.915p\_j^\{(T)\}\; =\; \backslash frac\{p\_j^\{1/T\}\}\{\backslash sum\_\{i\}\; p\_i^\{1/T\}\},\; \backslash qquad\; T^\backslash star\; =\; \backslash arg\backslash min\_\{T\}\; \backslash ;\; -\backslash sum\_\{(x,y)\; \backslash in\; \backslash mathcal\{C\}\}\; \backslash log\; p\_y^\{(T)\}(x)\; =\; 2.915$$

In plain words: the trained model was too sure of itself, so every percentage is softened by one number. That number is chosen on 60 messages the model never trained on, so the softened confidences match how often it is right.

Hand-over threshold. With confidence $c(x) = \max_j p_j^{(T)}(x)$, coverage and selective accuracy at a threshold $\tau$ are

$$\mathrm{c}\mathrm{o}\mathrm{v}(\tau )=\mathrm{Pr}[c(x)\ge \tau ],\mathrm{a}\mathrm{c}\mathrm{c}(\tau )=\mathrm{Pr}[\widehat{y}(x)=y\mid c(x)\ge \tau ]\backslash mathrm\{cov\}(\backslash tau)\; =\; \backslash Pr\backslash left[c(x)\; \backslash ge\; \backslash tau\backslash right],\; \backslash qquad\; \backslash mathrm\{acc\}(\backslash tau)\; =\; \backslash Pr\backslash left[\backslash hat\; y(x)\; =\; y\; \backslash mid\; c(x)\; \backslash ge\; \backslash tau\backslash right]$$

and the threshold is the most permissive one that keeps the target accuracy on the 150-message development split: $\tau^\star = \min {\tau : \mathrm{acc}(\tau) \ge 0.95} = 0.783$. Lowering $\tau$ routes more messages automatically at lower accuracy; the curve above plots $(\mathrm{cov}(\tau), \mathrm{acc}(\tau))$ on the holdout, with an exact (Clopper-Pearson) 95% interval for each point as the shaded band.

In plain words: if the model is at least 78.3% sure, it routes the message; otherwise a person takes it. That cut-off was chosen on separate messages as the lowest one that still kept automatic decisions 95% correct.

Metrics. Over $N$ messages with one-hot truth $y$: accuracy; multiclass Brier $\frac{1}{N}\sum_n \sum_j (p_{nj} - y_{nj})^2$; log loss $-\frac{1}{N}\sum_n \log p_{n,y_n}$; expected calibration error over ten confidence bins $\sum_b \frac{|B_b|}{N} \left|\mathrm{acc}(B_b) - \mathrm{conf}(B_b)\right|$; and order consistency, the share of messages whose chosen team survives a reshuffle of the options. Confidence intervals come from a paired bootstrap that resamples message families, so near-duplicate phrasings cannot inflate certainty.

In plain words: besides plain accuracy, we measure whether the confidence is honest (Brier, calibration error), whether the answer changes when the options are reordered, and how wide the uncertainty is, by resampling the test messages thousands of times.

Training

Full fine-tune of Qwen3.5-0.8B, a hybrid model with linear-attention layers. Only the answer position's logits are materialised (logits_to_keep=1), so each step costs one forward pass over the prompt and a six-way softmax.

Setting	Value
Parameters updated	all (full fine-tune), bf16
Optimizer	AdamW, learning rate 2e-5, cosine decay, 3% warmup, no weight decay, gradient clip 1.0
Data	2,076 training rows (each message in 2 orderings), 234 validation rows
Batch	effective 16; one unpadded example per forward pass, since left padding and linear-attention layers do not mix
Schedule	3 epochs, 390 optimizer steps
Hardware	one NVIDIA A40, about 29 minutes
Loss	1.93 at the start, 0.13 at the end (25-step mean)

Results

Sealed holdout: 90 messages, 15 per team, 30 of them touching two or more teams.

	Accuracy	Brier ↓	Log loss ↓	Routed automatically	Accuracy of those
Qwen3.5-0.8B, same prompt, no training	55.6%	0.581	1.172	3.3%	100%
tr-telekom-router	93.3%	0.113	0.265	84.4%	98.7%
Teacher model	100%	–	–	–	–

• +37.8 points over the untuned model, 95% CI [26.7, 48.9].
• 14 of 90 messages handed to a human; 1 of the 76 automatic decisions was wrong (98.7% correct, exact 95% interval 92.9% to 100%). 5 of the model's 6 mistakes had a confidence below the threshold and went to a human.
• Per team: İptal ve şikayet and Tarife ve paket 100%; Fatura, Cihaz and Numara taşıma 93.3%; Teknik arıza 80.0%.
• Single-team messages 96.7%, multi-team messages 86.7%.
• On the development split the model held 86.3% accuracy when class shares were shifted toward billing and fault messages, as in a real call centre.

The six teams

Team id	Team	Owns
fatura	Fatura	bill amount, payment, debt, auto-payment, refund, bill dispute
tarife_paket	Tarife ve paket	plan change, add-on GB/minutes, quota, campaigns
teknik_ariza	Teknik arıza	no signal, slow/no internet, dropped calls, modem/home internet faults
numara_tasima	Numara taşıma	porting a number in or out, porting status
cihaz	Cihaz	device purchase and installments, SIM/eSIM, device setup, broken purchased device
iptal_sikayet	İptal ve şikayet	cancellation, commitment exit fee, staff/service complaints, legal threats

When a message touches several teams, seven precedence rules in system_prompt.txt decide. For example, an explicit wish to cancel or a legal threat goes to İptal ve şikayet even when the cause is a bill, and moving the number to another operator goes to Numara taşıma even when the customer is leaving.

How it decides

flowchart LR
  M[customer message] --> P["system rules + message + options A–F"]
  P --> A["answer opened with 'Cevap: ('"]
  A --> L["probabilities of A–F, one forward pass"]
  L --> S["temperature 2.915"]
  S --> D{"top ≥ 0.783?"}
  D -- yes --> T[route to that team]
  D -- no --> H[hand to a human]

Usage

decision_config.json holds the option letters, the prompt template, the temperature and the threshold.

import json
import math
import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

path = snapshot_download("eqlx-ai/tr-telekom-router")
cfg = json.load(open(f"{path}/decision_config.json"))
system = open(f"{path}/system_prompt.txt").read()
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype="auto").eval()
letters = [o["letter"] for o in cfg["options"]]


def route(message: str) -> dict:
    options = "\n".join(f"{o['letter']}) {o['label']}: {o['description']}" for o in cfg["options"])
    user = cfg["prompt_template"].format(message=message, question=cfg["question"], options=options)
    msgs = [{"role": "system", "content": system}, {"role": "user", "content": user},
            {"role": "assistant", "content": cfg["assistant_prefix"]}]
    ids = tok.apply_chat_template(msgs, continue_final_message=True, add_generation_prompt=False,
                                  enable_thinking=False, return_dict=True, return_tensors="pt")["input_ids"]
    with torch.no_grad():
        logp = torch.log_softmax(model(input_ids=ids.to(model.device)).logits[0, -1].float(), dim=-1)
    # The readout the temperature and threshold were measured with: the top 20 tokens, letter variants
    # ("A", " A", "(A") summed, renormalised over the six letters; a letter outside the top 20 counts as 1e-12.
    top = torch.topk(logp, 20)
    mass = dict.fromkeys(letters, 0.0)
    for lp, i in zip(top.values.tolist(), top.indices.tolist()):
        t = tok.decode([i]).strip().lstrip("(").strip()
        if t in mass:
            mass[t] += math.exp(lp)
    total = sum(mass.values()) or 1.0
    w = [max(mass[l] / total, 1e-12) ** (1.0 / cfg["temperature"]) for l in letters]
    probs = [x / sum(w) for x in w]
    best = max(range(len(probs)), key=probs.__getitem__)
    team = cfg["options"][best]["id"]
    return {"team": team if probs[best] >= cfg["handover_threshold"] else "human",
            "confidence": round(probs[best], 3),
            "probs": {o["id"]: round(p, 3) for o, p in zip(cfg["options"], probs)}}


print(route("faturam bu ay 480 tl gelmiş normalde 250 oluyo"))

With vLLM, send the same three messages to /v1/chat/completions with max_tokens: 1, logprobs: true, top_logprobs: 20, continue_final_message: true, add_generation_prompt: false and chat_template_kwargs: {"enable_thinking": false}, then apply the same readout to the returned top-20 list. Keep the system prompt, the option order and the question exactly as in decision_config.json: they are part of what the model was trained on.

Why the top-20 readout matters. The temperature and the threshold were fitted on probabilities read from the top 20 tokens. When a letter falls outside the top 20 it counts as zero, and a temperature cannot soften a zero, so this readout gives higher confidences than reading all six letter logits directly. The choice of team is the same either way; only the confidence differs. If you read the six logits directly, refit the temperature and the threshold on your own labelled messages before using the 0.783 cut-off.

Engineering notes

• Why not just prompt a bigger model? Qwen3.5-4B with six worked examples in the prompt reached 85.3% on the development split: five times the size, and still below this 0.8B model on the same split (88.0%).
• More data is not always better. A second round with 590 extra multi-team messages and a second teacher gave the same accuracy and, after the longer full fine-tune, damaged the model's general chat ability. It was measured, rejected and is not this checkpoint.
• Reproducibility. The same recipe gave 91.3%, 87.3% and 88.0% on the development split across three training runs. This checkpoint is the 88.0% run; the holdout result above is its own measurement.

Limitations

• Training and test messages are synthetic, written by language models. Validate on a labelled sample of your own traffic, and refit the temperature and threshold there, before production use.
• Multi-team messages are the weakest group (86.7%), and an error there can come with high confidence.
• Trained for these six teams and these precedence rules; changing them means retraining.
• A routing model, not a chat model: general question answering dropped from 55.4% to 49.6% after training.

Licence and provenance

Fine-tuned from Qwen3.5-0.8B and released under Apache-2.0. The training messages and their labels were produced by larger language models accessed through commercial APIs, under those providers' terms. No real customer data was used. Built by eqlx.ai.