Repository navigation
Bantu ITN: shared kernel with a test-only language profile - #518
Conversation
Adds nemo_text_processing/inverse_text_normalization/bantu/, a shared inverse text normalization kernel that Bantu languages configure with a language profile. It supports cardinal and money, resolved in a sentence-level classifier; the verbalizers only format. The kernel is not listed in the language codes. Tests build the classifier and verbalizer directly with a made-up test language (currency code XTS), run through NeMo's inverse_normalize runtime, and check that every test input has exactly one best reading. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
- Pass any other word (capitals, digits, punctuation, any script) through
unchanged and collapse repeated spaces, instead of failing on such input.
- Split the taggers into CardinalFst, MoneyFst and WordFst classes, with
the place-value builders in number.py.
- Move test cases into test_cases_*.txt files read by parse_test_case_file.
- Add a Jenkinsfile stage for the kernel tests.
- Use typing.Optional/Tuple, NeMo's logger and the {lang}_itn_lower_cased.far
cache name; move the vowel list into the profile.
- Fix the money-signal docstring and remove project-internal notes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
The test language now has a ten-thousands number place and counts a listed noun with its ten-thousands scale, as for millions, so a count of 10 or more after a listed noun is read like after any other word. Corrects the comment on how listed nouns are counted and adds test cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
tbartley94
left a comment
There was a problem hiding this comment.
Minor changes needed:
- Expansion of cardinal cases
- attribution for crane labs (optional)
- Left questions about reasoning for semiotic class distinctions. Not necessarily a problem but curious about justification.
I'm also uncertain if the profile approach should be used. It's rather clean but departs from our normal convention so adds maintenance overhead. Any further explanation on the approach?
| @@ -0,0 +1,88 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
why is this distinct from a cardinal?
There was a problem hiding this comment.
number.py holds the place-value builders that cardinal, money and scale all use. CardinalFst can't be the shared piece as it stands, because it also decides the context (the noun before the number). Other languages pass a CardinalFst into MoneyFst and reuse its graph, and we can follow that: move these builders into CardinalFst and have MoneyFst take the cardinal. I'd do that as part of the restructure in my main comment.
There was a problem hiding this comment.
see suggestion for scale. non-primary semiotic classes we can contain in a modules folder.
There was a problem hiding this comment.
Done in 0c617e8. number.py is gone: its builders are in taggers/cardinal.py, the digit helpers in graph_utils.py, and MoneyFst takes the CardinalFst.
- add Crane Labs to the licence headers - move taggers/sentence.py to bantu/graph_utils.py - use NEMO_SPACE, string.digits and GraphFst.add_tokens - move the thousands separator into the profile - cardinal cases: enumerate 1-20, add million-range samples - money cases: base cases for USD, GBP, EUR and JPY Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
|
Thanks for the comments, @tbartley94. I've pushed d2752d1 with the smaller fixes and replied to the questions inline. On the profile: it was simply the quickest way for each language to hand its words and settings to the shared kernel. You're right that it departs from the usual interface: the profile argument on the constructors, a language written in Python instead of data files, and the test helper that builds the normalizer by hand. Proposal: keep the kernel's builders in
Adding a Bantu language would then mean adding its TSVs and tests. Does this work for you? If so, I'll restructure the PR in one pass. |
Approve of this approach. I do like the profile idea and asked a colleague to look over it for suggestion if we want to integrate into future PRs. So retain that code if you find it useful. But for now let's move forward in that direction. |
…es the cardinal Digit helpers (DIGIT, NONZERO, mapping, integer_domain) move to bantu/graph_utils.py; build_cardinal and build_role move into taggers/cardinal.py. CardinalFst keeps the agreeing graph per noun class and the role graphs, and MoneyFst builds its amounts from them. Output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
build_classifier(profile) and build_final_verbalizer(max_value, thousands_separator) replace the kernel's ClassifyFst and VerbalizeFinalFst classes, so each language keeps NeMo's standard constructors and calls these. format_integer moves to bantu/graph_utils.py. Output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
bantu/data_loader.py fills a language's profile skeleton (the shape of its grammar, written in settings.py) with the word lists in its data/*.tsv files. The test language's word lists move from tests/.../toy_profile.py to inverse_text_normalization/toy/data/, and its skeleton to toy/settings.py; the loaded profile equals the old one field by field. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
The test language 'toy' now has NeMo's standard ClassifyFst(cache_dir, whitelist, overwrite_cache, input_case) and VerbalizeFinalFst(), and is registered in InverseNormalizer but not offered on the command line. Its case tests move to tests/nemo_text_processing/toy/ and run through InverseNormalizer(lang='toy'); the build_normalizer test helper and toy_profile.py are removed. Kernel options that a case file cannot switch stay in bantu/test_kernel_options.py. Output is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
load_profile now rejects, with the file and line: empty fields, a currency whose class has no compounding numbers, and a currency role missing from roles.tsv. A missing data folder is an error instead of an empty language. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Cal3bdev <lwangacalebb@gmail.com>
|
Restructured as agreed, in ac6899c and the commits before it:
I've updated the PR description with the new layout and what adding a Bantu language involves. |
tbartley94
left a comment
There was a problem hiding this comment.
LGTM, going to run local. once tests pass i'll merge to staging for sparrowhawk tests. (also need to do some refactoring that I'll just take up on my end for moving from staging.)
| super().__init__(name="money", kind="verbalize") | ||
| number = format_integer(max_value, thousands_separator) | ||
| code = pynini.closure(NEMO_UPPER, 3, 3) | ||
| graph = ( |
There was a problem hiding this comment.
why can't you use the delete tokens function here as in other languages?
| from nemo_text_processing.text_normalization.en.graph_utils import GraphFst, delete_space | ||
|
|
||
|
|
||
| def format_integer(max_value: int, thousands_separator: str) -> "pynini.FstLike": |
There was a problem hiding this comment.
yes this is more a graph_utils function so we can keep there.
| counted_scales=tuple(replace(s, implied_currency="") for s in TOY_PROFILE.counted_scales), | ||
| ) | ||
| normalizer = build_normalizer(plain) | ||
| assert normalizer.inverse_normalize("mina epi", verbose=False) == "mina epi" |
There was a problem hiding this comment.
let's go with case file just to keep things a bit flatter. keeping regularity across prexisting languages help with maintenance.
| thousands_separator: written separator between digit groups | ||
| """ | ||
|
|
||
| def __init__(self, profile: CardinalProfile, thousands_separator: str = ","): |
There was a problem hiding this comment.
i'm still seeing it in cardinal and decimal fst construction. is there enough variance in the respective languages to need the balances? (like, is the bantu cline just completely inconsistent with which languages borrow the end stop and comma for separation?)
| @@ -0,0 +1,144 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
i gotcha, okay separation is fine. let's just not introduce a new semiotic class and we're fine.
| @@ -0,0 +1,144 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
actually, anything that's non-primary semiotic class, let's create a 'modules' subdirectory to contain them more consistent with our other repos and can help with once overs.
| @@ -0,0 +1,88 @@ | |||
| # Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
see suggestion for scale. non-primary semiotic classes we can contain in a modules folder.
What does this PR do ?
Adds a shared inverse text normalization (ITN) kernel for Bantu languages in
nemo_text_processing/inverse_text_normalization/bantu/, and a small made-up testlanguage,
toy, built on it. Luganda and Swahili will be built the same way: each is anormal NeMo language with its word lists in
data/*.tsv, calling the shared buildersinstead of writing its own grammars.
This is stage 1 of the staged plan agreed with @tbartley94: the kernel and a small
test-only language, with tests that check the kernel on its own. Luganda and Swahili
follow in their own PRs.
bantu/is shared code, not a language. The test languagetoyhas NeMo's standardClassifyFst(cache_dir, whitelist, overwrite_cache, input_case)andVerbalizeFinalFst(),and is registered in
InverseNormalizerbut hidden from the command line. Its tests runthrough
InverseNormalizer(lang='toy')like any other language.Collection: ITN
Why a shared kernel
Bantu languages share the hard parts of number reading:
different spoken forms (Luganda: abantu babiri "2 people", amagi abiri "2 eggs").
after amagi "eggs", but 20 after a currency word).
"twenty mu five" but also means "in").
"two ten-thousands" = 20,000).
The kernel handles these once; a language supplies only its words and settings.
How it works
then each run is read in the context of the word before it: a counted noun
(
CardinalFst), a currency phrase or money cue (MoneyFst), or nothing, in whichcase the words stay unchanged (
WordFst). Output is the standard token format:tokens { name: "bakopa" } tokens { cardinal { integer: "21" } }.(digit grouping,
<amount> <ISO code>).leaving words unchanged. Readings are made unique by structure, and every test also
checks that its input has exactly one best reading. (NeMo's runtime picks one reading
silently when there is a tie, so this check matters.)
punctuation, any script) passes through unchanged, and repeated spaces are collapsed.
Punctuation attached to a number word (
bapi.) leaves that number unchanged; fullcased and punctuation handling comes with each language.
cardinal,moneyand plain words.Suggested reading order
toy/settings.py: the test language's grammar shape, with the Luganda pattern eachfeature mirrors; then its word lists in
toy/data/.bantu/data_loader.py: how the TSVs fill the language's profile (bantu/profile.py,internal to the kernel).
toy/taggers/tokenize_and_classify.py, thenbantu/taggers/tokenize_and_classify.py(
build_classifier),cardinal.py,money.py,word.py,scale.py.bantu/graph_utils.py: digit helpers, number-word runs and the money-signal check.bantu/verbalizers/.Adding a Bantu language
A new folder like
toy/: itsdata/*.tsv, asettings.py(noun classes, join words,places, scales and their switches), the two short
ClassifyFst/VerbalizeFinalFstfiles, its tests, and one
elifinInverseNormalizer. No kernel code changes.Files
The test-only language
toyis a small made-up language. Every word is invented, but it is shapedlike a Bantu language so that each kernel feature is used: noun classes with agreement,
a join word that is also an ordinary word, a currency phrase that licenses a special
reading, ten-thousands and millions counted with scale nouns, implied currency with
money cues, contracted joins, and a word that continues a number but is not handled
yet. Its own currency code is
XTS, the ISO 4217 code reserved for testing; it also hasmade-up words for dollars, pounds, euros and yen (USD, GBP, EUR, JPY).
Because the words are invented, a failing test points at the kernel, not at real
language data.
Tests
bantu/andtoy/covered.test_cases_*.txt, as for other languages: countednumbers, money, words left unchanged, numbers inside sentences, malformed number
runs, and input with capitals, digits, punctuation and other scripts.
input_casevalues.Jenkinsfilestage runs these tests in CI.Outside this PR, we also ran our 205 Luganda development cases (cardinals and money)
through this kernel in NeMo's runtime: all pass, each with exactly one best reading. The
Luganda word lists also load unchanged from the same TSV layout.
That data will come with the Luganda PR after native-speaker review.
Questions for the reviewer
toyis not inpynini_export.py, since you're writing your ownscript for it. Happy to add an export entry if that's easier.
Not in this PR
its data and tests. Swahili test cases are being reviewed by Ernest Omondi.
Before your PR is "Ready for review"
Pre checks:
git commit -sto sign.pytestor (if your machine does not have GPU)pytest --cpufrom the root folder (given you marked your test cases accordingly@pytest.mark.run_only_on('CPU')).bash tools/text_processing_deployment/export_grammars.sh --MODE=test ...: next step, see abovepytestand Sparrowhawk here. pytest added; Sparrowhawk is the next step__init__.pyfor every folder and subfolder, includingdatafolder which has .TSV files?Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.to all newly added Python files? (year 2026)Copyright 2015 and onwards Google, Inc.. n/a,graph_utilsis imported, not copiedtry import: ... except: ...) if not already done. None presentPR Type:
If you haven't finished some of the above items you can still open "Draft" PR.
🤖 Generated with Claude Code