ghana-speech-phoneme-asr
A single CTC model that transcribes speech in 42 Ghanaian and West African languages directly into IPA phonemes, with punctuation.
Built for phoneme-level forced alignment and pronunciation work: the output is a sequence of IPA units rather than orthographic words, so it can be aligned to audio, compared across languages, or used as a TTS front-end target.
| Base | omniASR_W2V_300M encoder + a CTC head trained from scratch |
| Parameters | 316 M |
| Output | 172 IPA phoneme units + 4 special tokens |
| Training data | 2,329 h across 42 languages |
| Held-out UER | 16.64% (15.39% excluding punctuation) |
| Runtime | ONNX / sherpa-onnx, or PyTorch via fairseq2 |
Quick start (sherpa-onnx)
import sherpa_onnx, soundfile as sf
rec = sherpa_onnx.OfflineRecognizer.from_omnilingual_asr_ctc(
model="onnx/model.int8.onnx", # or model.onnx for fp32
tokens="onnx/tokens.txt",
num_threads=4,
)
wav, sr = sf.read("utterance.wav", dtype="float32") # 16 kHz mono
s = rec.create_stream()
s.accept_waveform(sr, wav)
rec.decode_stream(s)
print(s.result.tokens) # ['m', 'f', 'ia', 's', 'e', 'n', 'o', 'o', 'ɲ', ...]
print(s.result.text) # run together
result.tokens is the list you want — each element is one phoneme unit. Several units are
multi-character (kʰ, k͡p, t͡ʃ, nʷ), so never split the text on characters; that
would turn one sound into two.
Files
onnx/model.onnx fp32 ONNX, 1.3 GB
onnx/model.int8.onnx int8 quantised, 349 MB <- use this for deployment
onnx/tokens.txt 176 tokens, real IPA symbols
checkpoint/ fairseq2 checkpoint (model + optimizer + trainer state)
tokenizer/phoneme_spm.model SentencePiece char model over the phoneme proxies
tokenizer/phonemes.json proxy codepoint <-> IPA mapping, and unit frequencies
tokenizer/inventories.json each language's own phoneme inventory
config/ctc-ghana-ipa.yaml the training recipe config
config/model_arch.yaml model architecture
config/ghana_ipa.yaml fairseq2 asset cards (tokenizer + dataset)
config/language_distribution.tsv hours per language, drives temperature sampling
eval_results.json per-language evaluation
Results
Evaluated on held-out dev audio (100 clips per language) decoded through sherpa-onnx. UER is unit error rate over phonemes; the phonemes-only column strips punctuation from both sides.
| UER | phonemes only | |
|---|---|---|
| Weighted overall | 16.64% | 15.39% |
Best and worst languages, phonemes-only:
| best | worst | ||
|---|---|---|---|
Selee snw |
5.6% | Ninkare gur |
46.2% |
Paasaal sig |
6.7% | Hausa hau |
38.0% |
Gonja gjn |
7.2% | Kabiye kbp |
35.6% |
Gikyode acd |
8.1% | Konkomba xon |
28.0% |
Tem kdh |
8.8% | Nzema nzi |
25.8% |
Full per-language numbers are in eval_results.json.
Read these per-language, not as one number. The spread is large and does not track data volume — Dagaare has 15 h and scores 13.0%, Hausa has 152 h and scores 38.0%. Phonemisation consistency appears to matter more than corpus size.
Ninkare, Hausa and Kabiye are outliers worth treating with suspicion. All three had heavily repaired rule data upstream, and their targets may still be inconsistent, which would cap achievable accuracy regardless of the model. Verify before relying on them.
Punctuation
Punctuation is included in the output (27 of the 172 units). It costs 1.25 UER points overall, and the model gets it right about 62% of the time — it is clearly using pauses and intonation. Quotation marks are the least predictable, since nothing in the audio marks where a quote opens.
How the phoneme vocabulary works
Phoneme units are frequently multi-character (kʰ, k͡p, t͡ʃ). The tokenizer family used
by omniASR segments by character, which would split those apart, so each unit was mapped to
a single Private Use Area codepoint (U+E000…) for training. onnx/tokens.txt maps them back,
so the deployed model emits real IPA and nothing downstream ever sees the proxies.
tokenizer/phonemes.json holds the mapping if you need to regenerate or extend the vocabulary.
Per-language output restriction
The model shares one 172-unit inventory across all 42 languages, but any single language uses
only 25–52 of them (24% of the total on average). If you know the language in advance, you can
restrict the output to that language's inventory — tokenizer/inventories.json lists them —
by slicing the final projection before export. This makes it impossible for the model to emit
another language's phonemes.
We did not do this by default, because the shared inventory is what gives the low-resource languages their transfer benefit, and the model rarely strays out of inventory in practice.
Training
Trained with omnilingual-asr's
wav2vec2.asr recipe on a single H200.
- CTC head trained from scratch on
omniASR_W2V_300M(encoder not frozen) - lr 1e-5, bf16 autocast, ~480 s of audio per step
- Temperature-sampled across languages (
beta_language=0.5) so 15-hour languages are not swamped by 200-hour ones - Best checkpoint at step 20,000, selected on validation UER
UER progression: 100% → 45.5% (step 4k) → 27.6% (9k) → 22.1% (13.5k) → 18.88% (20k).
Note the 18.88% validation figure and the 16.64% evaluation figure differ because they use different dev sampling; the per-language table is the one to trust.
Resuming or fine-tuning
checkpoint/ contains model, optimizer and trainer state, so training can resume exactly.
The prepared training corpus is published separately as
ghananlpcommunity/ghana-speech-ipa-asr-ready —
already in the partitioned parquet layout the recipe expects.
Code and worked examples: GhanaNLP/ghana-phoneme-asr
Known limitations
- Digits are dropped, not verbalised. Numbers in the audio produce no phonemes.
- Language is not identified. The model transcribes phonetically without knowing which language it heard; pair it with a language ID model if you need that.
- Punctuation is approximate, especially quotation marks.
- Clips longer than 40 s are outside the range the base model accepts.
- 10 of the 42 languages use phoneme rules borrowed from a related language, because none
existed for them. They perform well, but the phoneme values are plausible rather than
authoritative. See the
tierfield in the phoneme dataset.
Sources
- Audio: ghananlpcommunity/ghana-speech
- Phoneme targets: ghananlpcommunity/ghana-speech-phonemes
- Phonemisation: GhanaNLP/ghana-g2p
- Base model: facebookresearch/omnilingual-asr (Apache-2.0)
Licence
CC BY-NC 4.0, following the source audio corpus.