Skip to content

Bantu ITN: shared kernel with a test-only language profile - #518

Merged
tbartley94 merged 9 commits into
NVIDIA:staging/bantu_itnfrom
cal3bdev:bantu-kernel
Nov 5, 2026
Merged

tbartley94 merged 9 commits into
NVIDIA:staging/bantu_itnfrom
cal3bdev:bantu-kernel

Conversation

@cal3bdev

@cal3bdev cal3bdev commented Oct 25, 2026 •

Copy link
Copy Markdown

What does this PR do ?

Adds a shared inverse text normalization (ITN) kernel for Bantu languages in
nemo_text_processing/inverse_text_normalization/bantu/, and a small made-up test
language, toy, built on it. Luganda and Swahili will be built the same way: each is a
normal NeMo language with its word lists in data/*.tsv, calling the shared builders
instead of writing its own grammars.

This is stage 1 of the staged plan agreed with @tbartley94: the kernel and a small
test-only language, with tests that check the kernel on its own. Luganda and Swahili
follow in their own PRs.

bantu/ is shared code, not a language. The test language toy has NeMo's standard
ClassifyFst(cache_dir, whitelist, overwrite_cache, input_case) and VerbalizeFinalFst(),
and is registered in InverseNormalizer but hidden from the command line. Its tests run
through InverseNormalizer(lang='toy') like any other language.

Collection: ITN

Why a shared kernel

Bantu languages share the hard parts of number reading:

  • Number words agree with the noun class of what is counted, so the same digit has
    different spoken forms (Luganda: abantu babiri "2 people", amagi abiri "2 eggs").
  • The same word can mean different numbers in different contexts (Luganda abiri is 2
    after amagi "eggs", but 20 after a currency word).
  • The join word between number parts is also an ordinary word (Luganda mu joins
    "twenty mu five" but also means "in").
  • Large amounts are counted with scale nouns (Luganda mitwalo ebiri,
    "two ten-thousands" = 20,000).

The kernel handles these once; a language supplies only its words and settings.

How it works

  • The classifier reads the whole sentence. Runs of number words are marked first,
    then each run is read in the context of the word before it: a counted noun
    (CardinalFst), a currency phrase or money cue (MoneyFst), or nothing, in which
    case the words stay unchanged (WordFst). Output is the standard token format:
    tokens { name: "bakopa" } tokens { cardinal { integer: "21" } }.
  • Values are resolved in the tagger; the verbalizer only formats them
    (digit grouping, <amount> <ISO code>).
  • No weights decide between readings. Weights only prefer a normalization over
    leaving words unchanged. Readings are made unique by structure, and every test also
    checks that its input has exactly one best reading. (NeMo's runtime picks one reading
    silently when there is a tie, so this check matters.)
  • Input: the kernel reads lower-cased words. Any other word (capitals, digits,
    punctuation, any script) passes through unchanged, and repeated spaces are collapsed.
    Punctuation attached to a number word (bapi.) leaves that number unchanged; full
    cased and punctuation handling comes with each language.
  • Supported classes in this PR: cardinal, money and plain words.

Suggested reading order

  1. toy/settings.py: the test language's grammar shape, with the Luganda pattern each
    feature mirrors; then its word lists in toy/data/.
  2. bantu/data_loader.py: how the TSVs fill the language's profile (bantu/profile.py,
    internal to the kernel).
  3. toy/taggers/tokenize_and_classify.py, then bantu/taggers/tokenize_and_classify.py
    (build_classifier), cardinal.py, money.py, word.py, scale.py.
  4. bantu/graph_utils.py: digit helpers, number-word runs and the money-signal check.
  5. bantu/verbalizers/.

Adding a Bantu language

A new folder like toy/: its data/*.tsv, a settings.py (noun classes, join words,
places, scales and their switches), the two short ClassifyFst / VerbalizeFinalFst
files, its tests, and one elif in InverseNormalizer. No kernel code changes.

Files

nemo_text_processing/inverse_text_normalization/bantu/      shared kernel (not a language)
  profile.py                  the profile type, internal to the kernel
  data_loader.py              load_profile: fills a language's settings from its TSVs
  graph_utils.py              digit helpers, number-word runs, token formats, money-signal check
  taggers/
    tokenize_and_classify.py  build_classifier(profile): reads the whole sentence
    cardinal.py               CardinalFst and the place-value builders
    money.py                  MoneyFst(cardinal): amounts after a currency phrase, implied currency
    word.py                   WordFst: words left unchanged
    scale.py                  scale nouns with a count (ten-thousands, millions)
  verbalizers/
    cardinal.py, money.py, word.py, verbalize.py, verbalize_final.py (build_final_verbalizer)
nemo_text_processing/inverse_text_normalization/toy/        the test language
  settings.py                 grammar shape: noun classes, joins, places, scales
  data/numbers/*.tsv, data/money/*.tsv
  taggers/tokenize_and_classify.py, verbalizers/verbalize_final.py
nemo_text_processing/inverse_text_normalization/inverse_normalize.py   'toy' branch (not in CLI choices)
tests/nemo_text_processing/toy/
  data_inverse_text_normalization/test_cases_{cardinal,money,word}.txt
  test_cardinal.py, test_money.py, test_word.py, test_verbalizer.py
tests/nemo_text_processing/bantu/
  test_kernel_options.py      options a case file cannot switch, caching, hidden CLI
  test_data_loader.py         the TSV loader, including malformed rows
Jenkinsfile                   stage: L1: Run Bantu kernel ITN tests

The test-only language

toy is a small made-up language. Every word is invented, but it is shaped
like a Bantu language so that each kernel feature is used: noun classes with agreement,
a join word that is also an ordinary word, a currency phrase that licenses a special
reading, ten-thousands and millions counted with scale nouns, implied currency with
money cues, contracted joins, and a word that continues a number but is not handled
yet. Its own currency code is XTS, the ISO 4217 code reserved for testing; it also has
made-up words for dollars, pounds, euros and yen (USD, GBP, EUR, JPY).

Because the words are invented, a failing test points at the kernel, not at real
language data.

Tests

pytest tests/nemo_text_processing/bantu tests/nemo_text_processing/toy --cpu
  • 155 tests, all passing in about 17 seconds, with every line of bantu/ and toy/ covered.
  • Spoken/written pairs are in test_cases_*.txt, as for other languages: counted
    numbers, money, words left unchanged, numbers inside sentences, malformed number
    runs, and input with capitals, digits, punctuation and other scripts.
  • The verbalizer is tested on its own, including rejecting invalid values.
  • The classifier's FAR cache is saved and restored with the same results, for both
    input_case values.
  • Invalid settings and malformed TSV rows are rejected with a clear error (file and line).
  • A new Jenkinsfile stage runs these tests in CI.

Outside this PR, we also ran our 205 Luganda development cases (cardinals and money)
through this kernel in NeMo's runtime: all pass, each with exactly one best reading. The
Luganda word lists also load unchanged from the same TSV layout.
That data will come with the Luganda PR after native-speaker review.

Questions for the reviewer

  1. Sparrowhawk: toy is not in pynini_export.py, since you're writing your own
    script for it. Happy to add an export entry if that's easier.

Not in this PR

  • Sparrowhawk tests: next step, as agreed (see the question above).
  • Languages: Luganda (stage 2) and Swahili (stage 3) come in their own PRs, each with
    its data and tests. Swahili test cases are being reviewed by Ernest Omondi.
  • Cased input and attached punctuation: with each language.
  • Other classes (e.g. decimals): added later per language, updating the kernel as needed.

Before your PR is "Ready for review"

Pre checks:

  • Have you signed your commits? Use git commit -s to sign.
  • Do all unittests finish successfully before sending PR?
    1. pytest or (if your machine does not have GPU) pytest --cpu from the root folder (given you marked your test cases accordingly @pytest.mark.run_only_on('CPU')).
    2. Sparrowhawk tests bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...: next step, see above
  • If you are adding a new feature: Have you added test cases for both pytest and Sparrowhawk here. pytest added; Sparrowhawk is the next step
  • Have you added __init__.py for every folder and subfolder, including data folder which has .TSV files?
  • Have you followed codeQL results and removed unused variables and imports (report is at the bottom of the PR in github review box) ?
  • Have you added the correct license header Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. to all newly added Python files? (year 2026)
  • If you copied nemo_text_processing/text_normalization/en/graph_utils.py your header's second line should be Copyright 2015 and onwards Google, Inc.. n/a, graph_utils is imported, not copied
  • Remove import guards (try import: ... except: ...) if not already done. None present
  • If you added a new language or a new feature please update the NeMo documentation (lives in different repo). With the first language (Luganda), since the kernel is not selectable on its own
  • Have you added your language support to tools/text_processing_deployment/pynini_export.py. With the first language; see the question above

PR Type:

  • New Feature
  • Bugfix
  • Documentation
  • Test

If you haven't finished some of the above items you can still open "Draft" PR.

🤖 Generated with Claude Code

cal3bdev and others added 3 commits October 25, 2026 10:56
Adds nemo_text_processing/inverse_text_normalization/bantu/, a shared inverse
text normalization kernel that Bantu languages configure with a language
profile. It supports cardinal and money, resolved in a sentence-level
classifier; the verbalizers only format. The kernel is not listed in the
language codes.

Tests build the classifier and verbalizer directly with a made-up test
language (currency code XTS), run through NeMo's inverse_normalize runtime,
and check that every test input has exactly one best reading.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
- Pass any other word (capitals, digits, punctuation, any script) through
  unchanged and collapse repeated spaces, instead of failing on such input.
- Split the taggers into CardinalFst, MoneyFst and WordFst classes, with
  the place-value builders in number.py.
- Move test cases into test_cases_*.txt files read by parse_test_case_file.
- Add a Jenkinsfile stage for the kernel tests.
- Use typing.Optional/Tuple, NeMo's logger and the {lang}_itn_lower_cased.far
  cache name; move the vowel list into the profile.
- Fix the money-signal docstring and remove project-internal notes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
The test language now has a ten-thousands number place and counts a listed
noun with its ten-thousands scale, as for millions, so a count of 10 or
more after a listed noun is read like after any other word. Corrects the
comment on how listed nouns are counted and adds test cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
@cal3bdev
cal3bdev marked this pull request as ready for review October 27, 2026 06:09
@tbartley94
tbartley94 self-requested a review October 28, 2026 17:29

@tbartley94 tbartley94 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor changes needed:

  • Expansion of cardinal cases
  • attribution for crane labs (optional)
  • Left questions about reasoning for semiotic class distinctions. Not necessarily a problem but curious about justification.

I'm also uncertain if the profile approach should be used. It's rather clean but departs from our normal convention so adds maintenance overhead. Any further explanation on the approach?

@@ -0,0 +1,88 @@
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this distinct from a cardinal?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

number.py holds the place-value builders that cardinal, money and scale all use. CardinalFst can't be the shared piece as it stands, because it also decides the context (the noun before the number). Other languages pass a CardinalFst into MoneyFst and reuse its graph, and we can follow that: move these builders into CardinalFst and have MoneyFst take the cardinal. I'd do that as part of the restructure in my main comment.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see suggestion for scale. non-primary semiotic classes we can contain in a modules folder.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 0c617e8. number.py is gone: its builders are in taggers/cardinal.py, the digit helpers in graph_utils.py, and MoneyFst takes the CardinalFst.

- add Crane Labs to the licence headers
- move taggers/sentence.py to bantu/graph_utils.py
- use NEMO_SPACE, string.digits and GraphFst.add_tokens
- move the thousands separator into the profile
- cardinal cases: enumerate 1-20, add million-range samples
- money cases: base cases for USD, GBP, EUR and JPY

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
@cal3bdev

Copy link
Copy Markdown
Author

Thanks for the comments, @tbartley94. I've pushed d2752d1 with the smaller fixes and replied to the questions inline.

On the profile: it was simply the quickest way for each language to hand its words and settings to the shared kernel. You're right that it departs from the usual interface: the profile argument on the constructors, a language written in Python instead of data files, and the test helper that builds the normalizer by hand.

Proposal: keep the kernel's builders in bantu/, and make each language a normal NeMo language around them:

  • Each language (lg/, sw/) has the standard ClassifyFst(cache_dir, overwrite_cache, input_case) and VerbalizeFinalFst(), keeps its word lists in data/*.tsv, and is registered like any other language.
  • It reads its TSVs and calls the shared builders directly, with no inheritance. The profile becomes an internal detail.
  • The test language gets the same layout and is hidden from the CLI, as you suggested. That removes the build_normalizer helper and the profile arguments.
  • MoneyFst takes the CardinalFst, as in other languages.

Adding a Bantu language would then mean adding its TSVs and tests. Does this work for you? If so, I'll restructure the PR in one pass.

@tbartley94

Copy link
Copy Markdown
Member

Thanks for the comments, @tbartley94. I've pushed d2752d1 with the smaller fixes and replied to the questions inline.

On the profile: it was simply the quickest way for each language to hand its words and settings to the shared kernel. You're right that it departs from the usual interface: the profile argument on the constructors, a language written in Python instead of data files, and the test helper that builds the normalizer by hand.

Proposal: keep the kernel's builders in bantu/, and make each language a normal NeMo language around them:

* Each language (`lg/`, `sw/`) has the standard `ClassifyFst(cache_dir, overwrite_cache, input_case)` and `VerbalizeFinalFst()`, keeps its word lists in `data/*.tsv`, and is registered like any other language.

* It reads its TSVs and calls the shared builders directly, with no inheritance. The profile becomes an internal detail.

* The test language gets the same layout and is hidden from the CLI, as you suggested. That removes the `build_normalizer` helper and the profile arguments.

* `MoneyFst` takes the `CardinalFst`, as in other languages.

Adding a Bantu language would then mean adding its TSVs and tests. Does this work for you? If so, I'll restructure the PR in one pass.

Approve of this approach. I do like the profile idea and asked a colleague to look over it for suggestion if we want to integrate into future PRs. So retain that code if you find it useful. But for now let's move forward in that direction.

cal3bdev and others added 5 commits November 5, 2026 13:30
…es the cardinal

Digit helpers (DIGIT, NONZERO, mapping, integer_domain) move to bantu/graph_utils.py;
build_cardinal and build_role move into taggers/cardinal.py. CardinalFst keeps the
agreeing graph per noun class and the role graphs, and MoneyFst builds its amounts
from them. Output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
build_classifier(profile) and build_final_verbalizer(max_value, thousands_separator)
replace the kernel's ClassifyFst and VerbalizeFinalFst classes, so each language
keeps NeMo's standard constructors and calls these. format_integer moves to
bantu/graph_utils.py. Output is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
bantu/data_loader.py fills a language's profile skeleton (the shape of its grammar,
written in settings.py) with the word lists in its data/*.tsv files. The test
language's word lists move from tests/.../toy_profile.py to
inverse_text_normalization/toy/data/, and its skeleton to toy/settings.py; the
loaded profile equals the old one field by field.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
The test language 'toy' now has NeMo's standard ClassifyFst(cache_dir, whitelist,
overwrite_cache, input_case) and VerbalizeFinalFst(), and is registered in
InverseNormalizer but not offered on the command line. Its case tests move to
tests/nemo_text_processing/toy/ and run through InverseNormalizer(lang='toy');
the build_normalizer test helper and toy_profile.py are removed. Kernel options
that a case file cannot switch stay in bantu/test_kernel_options.py. Output is
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
load_profile now rejects, with the file and line: empty fields, a currency whose
class has no compounding numbers, and a currency role missing from roles.tsv. A
missing data folder is an error instead of an empty language.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
@cal3bdev

cal3bdev commented Nov 5, 2026

Copy link
Copy Markdown
Author

Restructured as agreed, in ac6899c and the commits before it:

  • toy is now a standard NeMo language: ClassifyFst(cache_dir, whitelist, overwrite_cache, input_case) and VerbalizeFinalFst(), words in toy/data/*.tsv, registered in InverseNormalizer but hidden from the CLI.
  • It calls the shared builders in bantu/, with no inheritance. The profile code is kept inside bantu/, and bantu/data_loader.py fills it from a language's TSVs.
  • MoneyFst takes the CardinalFst; number.py is folded into cardinal.py.
  • Output is unchanged: the old and new code give identical results on all test cases plus 12,000 generated inputs.

I've updated the PR description with the new layout and what adding a Bantu language involves. toy isn't in pynini_export.py, since you're writing your own Sparrowhawk script; happy to add an entry if that's easier.

@tbartley94 tbartley94 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, going to run local. once tests pass i'll merge to staging for sparrowhawk tests. (also need to do some refactoring that I'll just take up on my end for moving from staging.)

super().__init__(name="money", kind="verbalize")
number = format_integer(max_value, thousands_separator)
code = pynini.closure(NEMO_UPPER, 3, 3)
graph = (

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why can't you use the delete tokens function here as in other languages?

from nemo_text_processing.text_normalization.en.graph_utils import GraphFst, delete_space


def format_integer(max_value: int, thousands_separator: str) -> "pynini.FstLike":

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this is more a graph_utils function so we can keep there.

counted_scales=tuple(replace(s, implied_currency="") for s in TOY_PROFILE.counted_scales),
)
normalizer = build_normalizer(plain)
assert normalizer.inverse_normalize("mina epi", verbose=False) == "mina epi"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's go with case file just to keep things a bit flatter. keeping regularity across prexisting languages help with maintenance.

thousands_separator: written separator between digit groups
"""

def __init__(self, profile: CardinalProfile, thousands_separator: str = ","):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm still seeing it in cardinal and decimal fst construction. is there enough variance in the respective languages to need the balances? (like, is the bantu cline just completely inconsistent with which languages borrow the end stop and comma for separation?)

@@ -0,0 +1,144 @@
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i gotcha, okay separation is fine. let's just not introduce a new semiotic class and we're fine.

@@ -0,0 +1,144 @@
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

actually, anything that's non-primary semiotic class, let's create a 'modules' subdirectory to contain them more consistent with our other repos and can help with once overs.

@@ -0,0 +1,88 @@
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see suggestion for scale. non-primary semiotic classes we can contain in a modules folder.

@tbartley94
tbartley94 changed the base branch from main to staging/bantu_itn November 5, 2026 21:09
@tbartley94
tbartley94 merged commit 7e004a9 into NVIDIA:staging/bantu_itn Nov 5, 2026
3 checks passed
Comment on lines +100 to +103
from nemo_text_processing.inverse_text_normalization.bantu.taggers.scale import (
build_noun_scale_amount,
build_scale_amount,
)
NounContext,
agreement_context,
)
from nemo_text_processing.inverse_text_normalization.bantu.taggers.cardinal import build_cardinal
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants